ConcurrentRebalancersTest.TwoConcurrentRebalancers/1: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 3
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
W20240503 02:18:37.767304  8297 leader_election.cc:334] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:37.767989   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 7889
I20240503 02:18:37.774318  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e966caa651e4e42890be5d79f851a1f" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:18:37.779000  8295 leader_election.cc:302] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64; no voters: 2942efc0ecb448d2ba2f6bca20fbb29c, ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:18:37.779420 27613 raft_consensus.cc:2726] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:18:37.784482  8011 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:18:37.815774 27579 raft_consensus.cc:482] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca05c9a8bee148dab8a2a982b8fa2ea4)
I20240503 02:18:37.815982 27579 raft_consensus.cc:504] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:37.816603 27579 leader_election.cc:288] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
I20240503 02:18:37.817338  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e966caa651e4e42890be5d79f851a1f" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:18:37.817591  8361 raft_consensus.cc:2454] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 2.
I20240503 02:18:37.818086  8168 leader_election.cc:302] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64; no voters: 
I20240503 02:18:37.818729 27264 raft_consensus.cc:2781] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:18:37.818851 27264 raft_consensus.cc:482] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Starting leader election (detected failure of leader ca05c9a8bee148dab8a2a982b8fa2ea4)
I20240503 02:18:37.818929 27264 raft_consensus.cc:3037] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:18:37.819406  8167 leader_election.cc:334] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:37.822176 27264 raft_consensus.cc:504] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:37.823333 27264 leader_election.cc:288] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 election: Requested vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
I20240503 02:18:37.823325  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e966caa651e4e42890be5d79f851a1f" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:18:37.823539  8361 raft_consensus.cc:3037] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:18:37.825385  8361 raft_consensus.cc:2454] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 3.
I20240503 02:18:37.826002  8168 leader_election.cc:302] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64; no voters: 
I20240503 02:18:37.826459 27264 raft_consensus.cc:2781] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 3 FOLLOWER]: Leader election won for term 3
W20240503 02:18:37.826435  8167 leader_election.cc:334] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:37.826682 27264 raft_consensus.cc:686] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 3 LEADER]: Becoming Leader. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Running, Role: LEADER
I20240503 02:18:37.826948 27264 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:37.830108  7830 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: term changed from 2 to 3, leader changed from ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) to 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3). New cstate: current_term: 3 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:37.937268 27643 raft_consensus.cc:482] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d826d4dae2aa4e05b2f64c9b17a1126f)
I20240503 02:18:37.937494 27643 raft_consensus.cc:504] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:37.938174 27643 leader_election.cc:288] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:18:37.939014  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:18:37.939245  8232 raft_consensus.cc:2454] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 2.
W20240503 02:18:37.939647  8035 leader_election.cc:334] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111)
I20240503 02:18:37.939870  8035 leader_election.cc:302] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:18:37.940114 27643 raft_consensus.cc:2781] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:18:37.940207 27643 raft_consensus.cc:482] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Starting leader election (detected failure of leader d826d4dae2aa4e05b2f64c9b17a1126f)
I20240503 02:18:37.940295 27643 raft_consensus.cc:3037] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:18:37.942202 27643 raft_consensus.cc:504] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:37.942761 27643 leader_election.cc:288] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 election: Requested vote from peers d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:18:37.943488  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:18:37.943679  8232 raft_consensus.cc:3037] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:18:37.943955  8035 leader_election.cc:334] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111)
I20240503 02:18:37.946627  8232 raft_consensus.cc:2454] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 3.
I20240503 02:18:37.947074  8035 leader_election.cc:302] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:18:37.947389 27643 raft_consensus.cc:2781] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:18:37.947794 27643 raft_consensus.cc:686] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 LEADER]: Becoming Leader. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:18:37.948326 27643 consensus_queue.cc:237] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:37.951486  7830 catalog_manager.cc:5484] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: term changed from 2 to 3, leader changed from d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) to 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2). New cstate: current_term: 3 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:18:37.957288  9931 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.037544  8295 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:38.077664  8101 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 02:18:38.078328 27613 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 02:18:38.085817  8297 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:38.104564  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(2448e12efbd44a90b1f8bce9519e73e0): 4581 bytes on disk
I20240503 02:18:38.105286  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(2448e12efbd44a90b1f8bce9519e73e0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 02:18:38.106302   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 7889
I20240503 02:18:38.139642  8232 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 3 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 2 index: 18. Preceding OpId from leader: term: 3 index: 19. (index mismatch)
I20240503 02:18:38.140209 27372 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20240503 02:18:38.168808  8168 connection.cc:535] server connection from 127.0.145.1:39931 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:18:38.169833   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 8020
W20240503 02:18:38.170588  8035 consensus_peers.cc:485] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Couldn't send request to peer d986fcfb926e47baa7d2d5b99f697334. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:37681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.171018  9929 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Couldn't send request to peer d986fcfb926e47baa7d2d5b99f697334. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:37681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.172186  8295 consensus_peers.cc:485] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 -> Peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Couldn't send request to peer d986fcfb926e47baa7d2d5b99f697334. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:37681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.176316  9931 consensus_peers.cc:462] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:38.178221  8140 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:18:38.210019  8167 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.366430  8165 consensus_peers.cc:485] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Couldn't send request to peer d986fcfb926e47baa7d2d5b99f697334. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:37681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.425710  8165 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:38.442193  8361 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 3 index: 16. (index mismatch)
I20240503 02:18:38.442801 27685 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240503 02:18:38.464989  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2448e12efbd44a90b1f8bce9519e73e0: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:18:38.517385   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 8020
W20240503 02:18:38.579738  8297 connection.cc:535] client connection to 127.0.145.2:35309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:38.579833  8167 connection.cc:535] client connection to 127.0.145.2:35309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:18:38.580118   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 8150
W20240503 02:18:38.580963  8297 consensus_peers.cc:485] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:35309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.581017  8167 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:35309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.581118  8167 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:35309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:38.586120  8270 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:18:38.659272  9931 consensus_peers.cc:462] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
W20240503 02:18:38.698714  8297 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:35309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.705195  9931 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:35309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.714738  8297 consensus_peers.cc:485] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:35309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.812536  8297 consensus_peers.cc:485] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:35309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:38.821333  7819 maintenance_manager.cc:419] P 5680e1b3edfe402b831ab3889e06b093: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 11218 bytes on disk
I20240503 02:18:38.826810  7818 maintenance_manager.cc:643] P 5680e1b3edfe402b831ab3889e06b093: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s	user 0.005s	sys 0.000s Metrics: {"dirs.queue_time_us":402,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":91,"thread_start_us":283,"threads_started":1}
W20240503 02:18:38.857219  8297 consensus_peers.cc:485] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:18:38.942924  8295 consensus_peers.cc:485] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:18:38.952539   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 8150
I20240503 02:18:39.008888   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 8280
W20240503 02:18:39.009984  9929 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): Couldn't send request to peer 2942efc0ecb448d2ba2f6bca20fbb29c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.3:38203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:39.015020  8401 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:18:39.029681  9929 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:18:39.103854  9931 consensus_peers.cc:462] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:39.205210 27372 raft_consensus.cc:482] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d986fcfb926e47baa7d2d5b99f697334)
I20240503 02:18:39.205422 27372 raft_consensus.cc:504] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:39.206076 27372 leader_election.cc:288] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
W20240503 02:18:39.207468  9929 leader_election.cc:334] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Network error: Client connection negotiation failed: client connection to 127.0.145.1:37681: connect: Connection refused (error 111)
W20240503 02:18:39.207659  9929 leader_election.cc:334] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): Network error: Client connection negotiation failed: client connection to 127.0.145.3:38203: connect: Connection refused (error 111)
I20240503 02:18:39.207759  9929 leader_election.cc:302] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b85e85e631ab4e04a86bfe5a9f0aa512; no voters: 2942efc0ecb448d2ba2f6bca20fbb29c, d986fcfb926e47baa7d2d5b99f697334
I20240503 02:18:39.208024 27372 raft_consensus.cc:2726] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:18:39.272837 27372 raft_consensus.cc:482] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d986fcfb926e47baa7d2d5b99f697334)
I20240503 02:18:39.273059 27372 raft_consensus.cc:504] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:18:39.273707 27372 leader_election.cc:288] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
W20240503 02:18:39.275278  9929 leader_election.cc:334] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Network error: Client connection negotiation failed: client connection to 127.0.145.1:37681: connect: Connection refused (error 111)
W20240503 02:18:39.275457  9929 leader_election.cc:334] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111)
I20240503 02:18:39.275555  9929 leader_election.cc:302] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b85e85e631ab4e04a86bfe5a9f0aa512; no voters: d826d4dae2aa4e05b2f64c9b17a1126f, d986fcfb926e47baa7d2d5b99f697334
I20240503 02:18:39.275791 27372 raft_consensus.cc:2726] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:18:39.292263  7817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 97 similar messages]
I20240503 02:18:39.318416 27675 raft_consensus.cc:482] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:18:39.318646 27675 raft_consensus.cc:504] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:18:39.319324 27675 leader_election.cc:288] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
I20240503 02:18:39.320792   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 8280
W20240503 02:18:39.375845  9930 connection.cc:535] server connection from 127.0.145.4:52991 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:39.375880  9932 connection.cc:535] client connection to 127.0.145.4:35377 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:39.376173  7815 connection.cc:535] server connection from 127.0.145.4:49523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:18:39.376392   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 9913
W20240503 02:18:39.377455  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): Couldn't send request to peer 8d7c037d50c241b484ee843b06313a64. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.4:35377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:39.377563  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): Couldn't send request to peer 8d7c037d50c241b484ee843b06313a64. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.4:35377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:39.382402 10035 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:18:39.522897  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2448e12efbd44a90b1f8bce9519e73e0: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:18:39.708173 27680 raft_consensus.cc:482] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d986fcfb926e47baa7d2d5b99f697334)
I20240503 02:18:39.708392 27680 raft_consensus.cc:504] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:18:39.710237 27680 leader_election.cc:288] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
I20240503 02:18:39.715214 27711 raft_consensus.cc:482] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d986fcfb926e47baa7d2d5b99f697334)
I20240503 02:18:39.715607   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 9913
I20240503 02:18:39.715416 27711 raft_consensus.cc:504] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:18:39.766579   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 7799
I20240503 02:18:39.777695  7859 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:18:40.038266   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 7799
2024-05-03T02:18:40Z chronyd exiting
I20240503 02:18:40.108736   580 test_util.cc:175] -----------------------------------------------
I20240503 02:18:40.108851   580 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0

Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN      ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 02:11:17.481397   580 test_util.cc:268] Using random seed: 1009062792
I20240503 02:11:17.492781   580 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:11:17.492869   580 ts_itest-base.cc:113] --------------
I20240503 02:11:17.492918   580 ts_itest-base.cc:114] 6 tablet servers
I20240503 02:11:17.492961   580 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:11:17.492998   580 ts_itest-base.cc:116] --------------
2024-05-03T02:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:11:17Z Disabled control of system clock
I20240503 02:11:17.518550   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.145.62:38925
--webserver_interface=127.0.145.62
--webserver_port=0
--builtin_ntp_servers=127.0.145.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.145.62:38925
--location_mapping_cmd=/tmp/dist-test-taskfpuZdt/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 02:11:17.843828   592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:17.844179   592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:17.844321   592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:17.851941   592 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:17.852115   592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:11:17.852241   592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:17.852324   592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:11:17.852397   592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:11:17.861814   592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskfpuZdt/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.0.145.62:38925
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.145.62:38925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.145.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:17.863219   592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:17.865032   592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:17.876472   597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:17.877173   600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:17.877606   598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:17.878383   592 server_base.cc:1034] running on GCE node
I20240503 02:11:17.879213   592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:17.880729   592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:17.881937   592 hybrid_clock.cc:648] HybridClock initialized: now 1714702277881892 us; error 39 us; skew 500 ppm
I20240503 02:11:17.882278   592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:17.888752   592 webserver.cc:460] Webserver started at http://127.0.145.62:38731/ using document root <none> and password file <none>
I20240503 02:11:17.889583   592 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:17.889714   592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:17.890030   592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:17.892742   592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "76b23b9ca28642a885167cbc65e2dc83"
format_stamp: "Formatted at 2024-05-03 02:11:17 on dist-test-slave-gq9h"
I20240503 02:11:17.893523   592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "76b23b9ca28642a885167cbc65e2dc83"
format_stamp: "Formatted at 2024-05-03 02:11:17 on dist-test-slave-gq9h"
I20240503 02:11:17.898967   592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240503 02:11:17.902423   606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:17.903899   592 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:11:17.904081   592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
uuid: "76b23b9ca28642a885167cbc65e2dc83"
format_stamp: "Formatted at 2024-05-03 02:11:17 on dist-test-slave-gq9h"
I20240503 02:11:17.904292   592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:17.921861   592 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:17.922763   592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:17.923075   592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:17.941339   592 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.62:38925
I20240503 02:11:17.941359   657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.62:38925 every 8 connection(s)
I20240503 02:11:17.942771   592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:11:17.946812   658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:17.947185   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 592
I20240503 02:11:17.947468   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:11:17.958362   658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83: Bootstrap starting.
I20240503 02:11:17.961308   658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:17.962301   658 log.cc:824] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:17.965466   658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83: No bootstrap required, opened a new log
I20240503 02:11:17.973405   658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b23b9ca28642a885167cbc65e2dc83" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 38925 } }
I20240503 02:11:17.973767   658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:17.973908   658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b23b9ca28642a885167cbc65e2dc83, State: Initialized, Role: FOLLOWER
I20240503 02:11:17.974519   658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b23b9ca28642a885167cbc65e2dc83" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 38925 } }
I20240503 02:11:17.974781   658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:17.974932   658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:17.975111   658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:17.977880   658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b23b9ca28642a885167cbc65e2dc83" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 38925 } }
I20240503 02:11:17.978356   658 leader_election.cc:302] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76b23b9ca28642a885167cbc65e2dc83; no voters: 
I20240503 02:11:17.978991   658 leader_election.cc:288] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:11:17.979163   663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:17.980649   663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [term 1 LEADER]: Becoming Leader. State: Replica: 76b23b9ca28642a885167cbc65e2dc83, State: Running, Role: LEADER
I20240503 02:11:17.981218   658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:17.981235   663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b23b9ca28642a885167cbc65e2dc83" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 38925 } }
I20240503 02:11:17.984221   664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76b23b9ca28642a885167cbc65e2dc83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b23b9ca28642a885167cbc65e2dc83" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 38925 } } }
I20240503 02:11:17.984529   664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:17.985162   665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76b23b9ca28642a885167cbc65e2dc83. Latest consensus state: current_term: 1 leader_uuid: "76b23b9ca28642a885167cbc65e2dc83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b23b9ca28642a885167cbc65e2dc83" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 38925 } } }
I20240503 02:11:17.985610   665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:17.986021   671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:17.994263   671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:18.003036   671 catalog_manager.cc:1347] Generated new cluster ID: 31d4ddd894df4a5b85355ff8da615a4a
I20240503 02:11:18.003168   671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:18.061482   671 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:18.062755   671 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:18.069954   671 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 76b23b9ca28642a885167cbc65e2dc83: Generated new TSK 0
I20240503 02:11:18.071028   671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:18.078569   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.1:0
--local_ip_for_outbound_sockets=127.0.145.1
--webserver_interface=127.0.145.1
--webserver_port=0
--tserver_master_addrs=127.0.145.62:38925
--builtin_ntp_servers=127.0.145.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:11:18.433215   682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:18.433557   682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:18.433714   682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:18.441246   682 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:18.441421   682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:18.441593   682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.1
I20240503 02:11:18.450645   682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.145.1
--webserver_port=0
--tserver_master_addrs=127.0.145.62:38925
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.1
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:18.452057   682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:18.453752   682 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:18.466131   687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:18.466729   690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:18.467208   688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:18.467865   682 server_base.cc:1034] running on GCE node
I20240503 02:11:18.468526   682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:18.469492   682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:18.470718   682 hybrid_clock.cc:648] HybridClock initialized: now 1714702278470682 us; error 28 us; skew 500 ppm
I20240503 02:11:18.471036   682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:18.473253   682 webserver.cc:460] Webserver started at http://127.0.145.1:36777/ using document root <none> and password file <none>
I20240503 02:11:18.473990   682 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:18.474108   682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:18.474421   682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:18.477113   682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e8d9a43983f2478ea7ababa09b230177"
format_stamp: "Formatted at 2024-05-03 02:11:18 on dist-test-slave-gq9h"
I20240503 02:11:18.477952   682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e8d9a43983f2478ea7ababa09b230177"
format_stamp: "Formatted at 2024-05-03 02:11:18 on dist-test-slave-gq9h"
I20240503 02:11:18.483173   682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240503 02:11:18.486500   696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:18.487963   682 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:11:18.488121   682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e8d9a43983f2478ea7ababa09b230177"
format_stamp: "Formatted at 2024-05-03 02:11:18 on dist-test-slave-gq9h"
I20240503 02:11:18.488306   682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:18.514561   682 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:18.515489   682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:18.515763   682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:18.516819   682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:18.518316   682 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:18.518411   682 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:18.518524   682 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:18.518620   682 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:18.550985   682 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.1:34401
I20240503 02:11:18.551100   808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.1:34401 every 8 connection(s)
I20240503 02:11:18.552462   682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:11:18.560218   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 682
I20240503 02:11:18.560436   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:11:18.564020   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.2:0
--local_ip_for_outbound_sockets=127.0.145.2
--webserver_interface=127.0.145.2
--webserver_port=0
--tserver_master_addrs=127.0.145.62:38925
--builtin_ntp_servers=127.0.145.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:18.565737   809 heartbeater.cc:347] Connected to a master server at 127.0.145.62:38925
I20240503 02:11:18.566112   809 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:18.567174   809 heartbeater.cc:511] Master 127.0.145.62:38925 requested a full tablet report, sending...
I20240503 02:11:18.632283   623 ts_manager.cc:194] Registered new tserver with Master: e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401)
I20240503 02:11:18.634069   623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.1:57615
W20240503 02:11:18.911548   813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:18.911867   813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:18.911963   813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:18.919631   813 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:18.919768   813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:18.919929   813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.2
I20240503 02:11:18.929324   813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.145.2
--webserver_port=0
--tserver_master_addrs=127.0.145.62:38925
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.2
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:18.930737   813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:18.932502   813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:18.945618   822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:18.946000   819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:18.946578   820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:18.947235   813 server_base.cc:1034] running on GCE node
I20240503 02:11:18.947957   813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:18.949177   813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:18.950428   813 hybrid_clock.cc:648] HybridClock initialized: now 1714702278950381 us; error 39 us; skew 500 ppm
I20240503 02:11:18.950836   813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:18.959133   813 webserver.cc:460] Webserver started at http://127.0.145.2:41375/ using document root <none> and password file <none>
I20240503 02:11:18.960309   813 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:18.960479   813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:18.960901   813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:18.964897   813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
format_stamp: "Formatted at 2024-05-03 02:11:18 on dist-test-slave-gq9h"
I20240503 02:11:18.966190   813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
format_stamp: "Formatted at 2024-05-03 02:11:18 on dist-test-slave-gq9h"
I20240503 02:11:18.974462   813 fs_manager.cc:691] Time spent creating directory manager: real 0.008s	user 0.005s	sys 0.001s
I20240503 02:11:18.979348   828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:18.980981   813 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:11:18.981164   813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
format_stamp: "Formatted at 2024-05-03 02:11:18 on dist-test-slave-gq9h"
I20240503 02:11:18.981384   813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:19.013607   813 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:19.014586   813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:19.014930   813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:19.015930   813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:19.017438   813 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:19.017546   813 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:19.017665   813 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:19.017740   813 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:19.050160   813 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.2:43485
I20240503 02:11:19.050208   940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.2:43485 every 8 connection(s)
I20240503 02:11:19.051817   813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:11:19.060691   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 813
I20240503 02:11:19.060878   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:11:19.064854   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.3:0
--local_ip_for_outbound_sockets=127.0.145.3
--webserver_interface=127.0.145.3
--webserver_port=0
--tserver_master_addrs=127.0.145.62:38925
--builtin_ntp_servers=127.0.145.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:19.071162   941 heartbeater.cc:347] Connected to a master server at 127.0.145.62:38925
I20240503 02:11:19.071491   941 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:19.072329   941 heartbeater.cc:511] Master 127.0.145.62:38925 requested a full tablet report, sending...
I20240503 02:11:19.120153   623 ts_manager.cc:194] Registered new tserver with Master: 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485)
I20240503 02:11:19.121426   623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.2:34605
W20240503 02:11:19.417901   944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:19.418203   944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:19.418315   944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:19.426165   944 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:19.426331   944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:19.426448   944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.3
I20240503 02:11:19.436249   944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.145.3
--webserver_port=0
--tserver_master_addrs=127.0.145.62:38925
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.3
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:19.437554   944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:19.439292   944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:19.451601   951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:19.452102   953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:19.452687   950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:19.453418   944 server_base.cc:1034] running on GCE node
I20240503 02:11:19.454010   944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:19.455024   944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:19.456211   944 hybrid_clock.cc:648] HybridClock initialized: now 1714702279456155 us; error 61 us; skew 500 ppm
I20240503 02:11:19.456552   944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:19.458878   944 webserver.cc:460] Webserver started at http://127.0.145.3:38783/ using document root <none> and password file <none>
I20240503 02:11:19.459637   944 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:19.459745   944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:19.460062   944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:19.462699   944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f5e38dcb24084a1e969f83bf34aef88f"
format_stamp: "Formatted at 2024-05-03 02:11:19 on dist-test-slave-gq9h"
I20240503 02:11:19.463493   944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f5e38dcb24084a1e969f83bf34aef88f"
format_stamp: "Formatted at 2024-05-03 02:11:19 on dist-test-slave-gq9h"
I20240503 02:11:19.469161   944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.001s
I20240503 02:11:19.472434   959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:19.473697   944 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:19.473846   944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f5e38dcb24084a1e969f83bf34aef88f"
format_stamp: "Formatted at 2024-05-03 02:11:19 on dist-test-slave-gq9h"
I20240503 02:11:19.474032   944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:19.498795   944 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:19.500105   944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:19.500378   944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:19.501304   944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:19.502826   944 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:19.502938   944 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:19.503053   944 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:19.503127   944 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:19.537221   944 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.3:35511
I20240503 02:11:19.537269  1071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.3:35511 every 8 connection(s)
I20240503 02:11:19.538498   944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:19.544859   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 944
I20240503 02:11:19.545030   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:11:19.548197   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.4:0
--local_ip_for_outbound_sockets=127.0.145.4
--webserver_interface=127.0.145.4
--webserver_port=0
--tserver_master_addrs=127.0.145.62:38925
--builtin_ntp_servers=127.0.145.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:19.554035  1072 heartbeater.cc:347] Connected to a master server at 127.0.145.62:38925
I20240503 02:11:19.554338  1072 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:19.555228  1072 heartbeater.cc:511] Master 127.0.145.62:38925 requested a full tablet report, sending...
I20240503 02:11:19.612534   623 ts_manager.cc:194] Registered new tserver with Master: f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:19.613507   623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.3:34927
I20240503 02:11:19.637208   809 heartbeater.cc:503] Master 127.0.145.62:38925 was elected leader, sending a full tablet report...
W20240503 02:11:19.902943  1075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:19.903260  1075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:19.903411  1075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:19.911422  1075 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:19.911571  1075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:19.911728  1075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.4
I20240503 02:11:19.921288  1075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.145.4
--webserver_port=0
--tserver_master_addrs=127.0.145.62:38925
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.4
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:19.922643  1075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:19.924332  1075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:19.936475  1081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:19.937018  1084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:19.937498  1082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:19.938490  1075 server_base.cc:1034] running on GCE node
I20240503 02:11:19.939165  1075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:19.940110  1075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:19.941354  1075 hybrid_clock.cc:648] HybridClock initialized: now 1714702279941304 us; error 37 us; skew 500 ppm
I20240503 02:11:19.942219  1075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:19.944701  1075 webserver.cc:460] Webserver started at http://127.0.145.4:37637/ using document root <none> and password file <none>
I20240503 02:11:19.945459  1075 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:19.945573  1075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:19.945884  1075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:19.948587  1075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "288540692e7b404991570d9ca16ab6c0"
format_stamp: "Formatted at 2024-05-03 02:11:19 on dist-test-slave-gq9h"
I20240503 02:11:19.949359  1075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "288540692e7b404991570d9ca16ab6c0"
format_stamp: "Formatted at 2024-05-03 02:11:19 on dist-test-slave-gq9h"
I20240503 02:11:19.954552  1075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240503 02:11:19.957880  1090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:19.959348  1075 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240503 02:11:19.959533  1075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "288540692e7b404991570d9ca16ab6c0"
format_stamp: "Formatted at 2024-05-03 02:11:19 on dist-test-slave-gq9h"
I20240503 02:11:19.959738  1075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:19.984586  1075 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:19.985461  1075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:19.985745  1075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:19.986783  1075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:19.988468  1075 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:19.988552  1075 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:19.988632  1075 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:19.988692  1075 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:20.024513  1075 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.4:32773
I20240503 02:11:20.024561  1202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.4:32773 every 8 connection(s)
I20240503 02:11:20.025871  1075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:11:20.028100   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 1075
I20240503 02:11:20.028264   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:11:20.031646   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.5:0
--local_ip_for_outbound_sockets=127.0.145.5
--webserver_interface=127.0.145.5
--webserver_port=0
--tserver_master_addrs=127.0.145.62:38925
--builtin_ntp_servers=127.0.145.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:20.041177  1203 heartbeater.cc:347] Connected to a master server at 127.0.145.62:38925
I20240503 02:11:20.041532  1203 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:20.042564  1203 heartbeater.cc:511] Master 127.0.145.62:38925 requested a full tablet report, sending...
I20240503 02:11:20.091434   623 ts_manager.cc:194] Registered new tserver with Master: 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:20.092716   623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.4:34171
I20240503 02:11:20.124298   941 heartbeater.cc:503] Master 127.0.145.62:38925 was elected leader, sending a full tablet report...
W20240503 02:11:20.384310  1206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:20.384613  1206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:20.384716  1206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:20.392701  1206 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:20.392854  1206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:20.393056  1206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.5
I20240503 02:11:20.402437  1206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.145.5
--webserver_port=0
--tserver_master_addrs=127.0.145.62:38925
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.5
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:20.403815  1206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:20.405560  1206 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:20.418911  1213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:20.419423  1212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:20.420171  1215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:20.420945  1206 server_base.cc:1034] running on GCE node
I20240503 02:11:20.421557  1206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:20.422506  1206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:20.423714  1206 hybrid_clock.cc:648] HybridClock initialized: now 1714702280423669 us; error 42 us; skew 500 ppm
I20240503 02:11:20.424060  1206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:20.426265  1206 webserver.cc:460] Webserver started at http://127.0.145.5:37851/ using document root <none> and password file <none>
I20240503 02:11:20.427067  1206 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:20.427186  1206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:20.427495  1206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:20.430078  1206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "572709bd9ae041a4a98029298a6f69c1"
format_stamp: "Formatted at 2024-05-03 02:11:20 on dist-test-slave-gq9h"
I20240503 02:11:20.430912  1206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "572709bd9ae041a4a98029298a6f69c1"
format_stamp: "Formatted at 2024-05-03 02:11:20 on dist-test-slave-gq9h"
I20240503 02:11:20.435961  1206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.000s	sys 0.004s
I20240503 02:11:20.439404  1221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:20.440708  1206 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:11:20.440882  1206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "572709bd9ae041a4a98029298a6f69c1"
format_stamp: "Formatted at 2024-05-03 02:11:20 on dist-test-slave-gq9h"
I20240503 02:11:20.441074  1206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:20.457635  1206 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:20.458505  1206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:20.458846  1206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:20.459847  1206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:20.461372  1206 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:20.461469  1206 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:20.461586  1206 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:20.461685  1206 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:20.497673  1206 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.5:41189
I20240503 02:11:20.497712  1333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.5:41189 every 8 connection(s)
I20240503 02:11:20.499166  1206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:20.503271   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 1206
I20240503 02:11:20.503484   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:11:20.506556   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.6:0
--local_ip_for_outbound_sockets=127.0.145.6
--webserver_interface=127.0.145.6
--webserver_port=0
--tserver_master_addrs=127.0.145.62:38925
--builtin_ntp_servers=127.0.145.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:20.513275  1334 heartbeater.cc:347] Connected to a master server at 127.0.145.62:38925
I20240503 02:11:20.513629  1334 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:20.515357  1334 heartbeater.cc:511] Master 127.0.145.62:38925 requested a full tablet report, sending...
I20240503 02:11:20.560427   623 ts_manager.cc:194] Registered new tserver with Master: 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189)
I20240503 02:11:20.561710   623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.5:39027
I20240503 02:11:20.616245  1072 heartbeater.cc:503] Master 127.0.145.62:38925 was elected leader, sending a full tablet report...
W20240503 02:11:20.859230  1337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:20.859529  1337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:20.859663  1337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:20.867666  1337 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:20.867794  1337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:20.867944  1337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.6
I20240503 02:11:20.877048  1337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.145.6
--webserver_port=0
--tserver_master_addrs=127.0.145.62:38925
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.6
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:20.878513  1337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:20.880283  1337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:20.893368  1346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:20.893908  1343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:20.894595  1344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:20.895255  1337 server_base.cc:1034] running on GCE node
I20240503 02:11:20.895923  1337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:20.896870  1337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:20.898082  1337 hybrid_clock.cc:648] HybridClock initialized: now 1714702280898034 us; error 42 us; skew 500 ppm
I20240503 02:11:20.898406  1337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:20.900732  1337 webserver.cc:460] Webserver started at http://127.0.145.6:44499/ using document root <none> and password file <none>
I20240503 02:11:20.901475  1337 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:20.901597  1337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:20.901908  1337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:20.904542  1337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "4fda3b525c234f8f88aaefd355e7274e"
format_stamp: "Formatted at 2024-05-03 02:11:20 on dist-test-slave-gq9h"
I20240503 02:11:20.905833  1337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "4fda3b525c234f8f88aaefd355e7274e"
format_stamp: "Formatted at 2024-05-03 02:11:20 on dist-test-slave-gq9h"
I20240503 02:11:20.911069  1337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240503 02:11:20.914646  1352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:20.916059  1337 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:11:20.916229  1337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4fda3b525c234f8f88aaefd355e7274e"
format_stamp: "Formatted at 2024-05-03 02:11:20 on dist-test-slave-gq9h"
I20240503 02:11:20.916440  1337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:20.937819  1337 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:20.938757  1337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:20.939059  1337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:20.939988  1337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:20.941637  1337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:20.941756  1337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:20.941872  1337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:20.941946  1337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:20.975589  1337 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.6:40135
I20240503 02:11:20.975648  1464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.6:40135 every 8 connection(s)
I20240503 02:11:20.976994  1337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:11:20.987022   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 1337
I20240503 02:11:20.987226   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:11:20.990386  1465 heartbeater.cc:347] Connected to a master server at 127.0.145.62:38925
I20240503 02:11:20.990772  1465 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:20.991698  1465 heartbeater.cc:511] Master 127.0.145.62:38925 requested a full tablet report, sending...
I20240503 02:11:21.037779   623 ts_manager.cc:194] Registered new tserver with Master: 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135)
I20240503 02:11:21.038975   623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.6:49399
I20240503 02:11:21.043990   580 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 02:11:21.064596   580 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 02:11:21.096021  1203 heartbeater.cc:503] Master 127.0.145.62:38925 was elected leader, sending a full tablet report...
I20240503 02:11:21.565065  1334 heartbeater.cc:503] Master 127.0.145.62:38925 was elected leader, sending a full tablet report...
I20240503 02:11:22.041843  1465 heartbeater.cc:503] Master 127.0.145.62:38925 was elected leader, sending a full tablet report...
I20240503 02:11:24.818533   623 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44878:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:24.866158   744 tablet_service.cc:1467] Processing CreateTablet for tablet 16a92cef3eb143a5abcceffbd23cad67 (DEFAULT_TABLE table=rebalance_test_table_0 [id=218dec51342141d9aa8539b6824bf608]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:24.867678   743 tablet_service.cc:1467] Processing CreateTablet for tablet d7f194bfa67c4aeea454483ecb13dcb3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=218dec51342141d9aa8539b6824bf608]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:24.868808   743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7f194bfa67c4aeea454483ecb13dcb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:24.869946  1138 tablet_service.cc:1467] Processing CreateTablet for tablet 16a92cef3eb143a5abcceffbd23cad67 (DEFAULT_TABLE table=rebalance_test_table_0 [id=218dec51342141d9aa8539b6824bf608]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:24.870636  1400 tablet_service.cc:1467] Processing CreateTablet for tablet 16a92cef3eb143a5abcceffbd23cad67 (DEFAULT_TABLE table=rebalance_test_table_0 [id=218dec51342141d9aa8539b6824bf608]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:24.871521  1138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a92cef3eb143a5abcceffbd23cad67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:24.871989  1400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a92cef3eb143a5abcceffbd23cad67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:24.875458   744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a92cef3eb143a5abcceffbd23cad67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:24.878492   876 tablet_service.cc:1467] Processing CreateTablet for tablet fb165523c1e849eeb8656e2fc23b31dd (DEFAULT_TABLE table=rebalance_test_table_0 [id=218dec51342141d9aa8539b6824bf608]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:24.880383   876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb165523c1e849eeb8656e2fc23b31dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:24.894306  1488 tablet_bootstrap.cc:492] T 16a92cef3eb143a5abcceffbd23cad67 P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:24.896734  1006 tablet_service.cc:1467] Processing CreateTablet for tablet d7f194bfa67c4aeea454483ecb13dcb3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=218dec51342141d9aa8539b6824bf608]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:24.898134  1006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7f194bfa67c4aeea454483ecb13dcb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:24.900067  1488 tablet_bootstrap.cc:654] T 16a92cef3eb143a5abcceffbd23cad67 P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:24.895390  1007 tablet_service.cc:1467] Processing CreateTablet for tablet fb165523c1e849eeb8656e2fc23b31dd (DEFAULT_TABLE table=rebalance_test_table_0 [id=218dec51342141d9aa8539b6824bf608]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:24.903946  1007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb165523c1e849eeb8656e2fc23b31dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:24.904819  1489 tablet_bootstrap.cc:492] T fb165523c1e849eeb8656e2fc23b31dd P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:24.908424  1488 log.cc:824] T 16a92cef3eb143a5abcceffbd23cad67 P 4fda3b525c234f8f88aaefd355e7274e: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:24.909137  1268 tablet_service.cc:1467] Processing CreateTablet for tablet d7f194bfa67c4aeea454483ecb13dcb3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=218dec51342141d9aa8539b6824bf608]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:24.912070  1489 tablet_bootstrap.cc:654] T fb165523c1e849eeb8656e2fc23b31dd P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:24.912967  1490 tablet_bootstrap.cc:492] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:24.913813  1268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7f194bfa67c4aeea454483ecb13dcb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:24.912324  1269 tablet_service.cc:1467] Processing CreateTablet for tablet fb165523c1e849eeb8656e2fc23b31dd (DEFAULT_TABLE table=rebalance_test_table_0 [id=218dec51342141d9aa8539b6824bf608]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:24.915133  1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb165523c1e849eeb8656e2fc23b31dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:24.916791  1490 tablet_bootstrap.cc:654] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:24.918107  1490 log.cc:824] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:24.920019  1489 log.cc:824] T fb165523c1e849eeb8656e2fc23b31dd P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:24.924530  1488 tablet_bootstrap.cc:492] T 16a92cef3eb143a5abcceffbd23cad67 P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:24.924870  1488 ts_tablet_manager.cc:1367] T 16a92cef3eb143a5abcceffbd23cad67 P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.031s	user 0.008s	sys 0.000s
I20240503 02:11:24.925202  1493 tablet_bootstrap.cc:492] T fb165523c1e849eeb8656e2fc23b31dd P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:24.928946  1493 tablet_bootstrap.cc:654] T fb165523c1e849eeb8656e2fc23b31dd P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:24.932178  1493 log.cc:824] T fb165523c1e849eeb8656e2fc23b31dd P f5e38dcb24084a1e969f83bf34aef88f: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:24.933255  1489 tablet_bootstrap.cc:492] T fb165523c1e849eeb8656e2fc23b31dd P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:24.933619  1489 ts_tablet_manager.cc:1367] T fb165523c1e849eeb8656e2fc23b31dd P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.029s	user 0.007s	sys 0.000s
I20240503 02:11:24.936436  1490 tablet_bootstrap.cc:492] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:24.936733  1490 ts_tablet_manager.cc:1367] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.024s	user 0.004s	sys 0.003s
I20240503 02:11:24.937528  1488 raft_consensus.cc:348] T 16a92cef3eb143a5abcceffbd23cad67 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:24.937991  1488 raft_consensus.cc:374] T 16a92cef3eb143a5abcceffbd23cad67 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:24.938107  1488 raft_consensus.cc:729] T 16a92cef3eb143a5abcceffbd23cad67 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:24.938956  1488 consensus_queue.cc:260] T 16a92cef3eb143a5abcceffbd23cad67 P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:24.939786  1492 tablet_bootstrap.cc:492] T 16a92cef3eb143a5abcceffbd23cad67 P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:24.942834  1489 raft_consensus.cc:348] T fb165523c1e849eeb8656e2fc23b31dd P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:24.943434  1492 tablet_bootstrap.cc:654] T 16a92cef3eb143a5abcceffbd23cad67 P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:24.943511  1489 raft_consensus.cc:374] T fb165523c1e849eeb8656e2fc23b31dd P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:24.943727  1489 raft_consensus.cc:729] T fb165523c1e849eeb8656e2fc23b31dd P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:24.944715  1492 log.cc:824] T 16a92cef3eb143a5abcceffbd23cad67 P 288540692e7b404991570d9ca16ab6c0: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:24.944784  1489 consensus_queue.cc:260] T fb165523c1e849eeb8656e2fc23b31dd P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:24.948099  1493 tablet_bootstrap.cc:492] T fb165523c1e849eeb8656e2fc23b31dd P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:24.948508  1493 ts_tablet_manager.cc:1367] T fb165523c1e849eeb8656e2fc23b31dd P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.024s	user 0.007s	sys 0.005s
I20240503 02:11:24.948518  1498 tablet_bootstrap.cc:492] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:24.954239  1490 raft_consensus.cc:348] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:24.955962  1489 ts_tablet_manager.cc:1398] T fb165523c1e849eeb8656e2fc23b31dd P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.022s	user 0.015s	sys 0.000s
I20240503 02:11:24.962000  1498 tablet_bootstrap.cc:654] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:24.962072  1493 raft_consensus.cc:348] T fb165523c1e849eeb8656e2fc23b31dd P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:24.962538  1493 raft_consensus.cc:374] T fb165523c1e849eeb8656e2fc23b31dd P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:24.962783  1490 raft_consensus.cc:374] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:24.962967  1490 raft_consensus.cc:729] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:24.963721  1498 log.cc:824] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:24.963907  1490 consensus_queue.cc:260] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:24.964617  1493 raft_consensus.cc:729] T fb165523c1e849eeb8656e2fc23b31dd P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:24.965524  1488 ts_tablet_manager.cc:1398] T 16a92cef3eb143a5abcceffbd23cad67 P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.041s	user 0.011s	sys 0.004s
I20240503 02:11:24.965912  1493 consensus_queue.cc:260] T fb165523c1e849eeb8656e2fc23b31dd P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:24.967377  1492 tablet_bootstrap.cc:492] T 16a92cef3eb143a5abcceffbd23cad67 P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:24.967718  1492 ts_tablet_manager.cc:1367] T 16a92cef3eb143a5abcceffbd23cad67 P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.028s	user 0.010s	sys 0.017s
I20240503 02:11:24.969084  1490 ts_tablet_manager.cc:1398] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.032s	user 0.012s	sys 0.002s
I20240503 02:11:24.973286  1490 tablet_bootstrap.cc:492] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:24.975183  1490 tablet_bootstrap.cc:654] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:24.978919  1492 raft_consensus.cc:348] T 16a92cef3eb143a5abcceffbd23cad67 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:24.979557  1492 raft_consensus.cc:374] T 16a92cef3eb143a5abcceffbd23cad67 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:24.979707  1492 raft_consensus.cc:729] T 16a92cef3eb143a5abcceffbd23cad67 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:24.980544  1492 consensus_queue.cc:260] T 16a92cef3eb143a5abcceffbd23cad67 P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
W20240503 02:11:24.981532  1466 tablet.cc:2369] T 16a92cef3eb143a5abcceffbd23cad67 P 4fda3b525c234f8f88aaefd355e7274e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:24.983328  1493 ts_tablet_manager.cc:1398] T fb165523c1e849eeb8656e2fc23b31dd P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.035s	user 0.013s	sys 0.000s
I20240503 02:11:24.983891  1493 tablet_bootstrap.cc:492] T d7f194bfa67c4aeea454483ecb13dcb3 P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:24.984213  1490 tablet_bootstrap.cc:492] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:24.984381  1490 ts_tablet_manager.cc:1367] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.000s
I20240503 02:11:24.985621  1492 ts_tablet_manager.cc:1398] T 16a92cef3eb143a5abcceffbd23cad67 P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.018s	user 0.016s	sys 0.000s
I20240503 02:11:24.986207  1498 tablet_bootstrap.cc:492] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:24.986495  1498 ts_tablet_manager.cc:1367] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.038s	user 0.003s	sys 0.014s
I20240503 02:11:24.985956  1490 raft_consensus.cc:348] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:24.986888  1490 raft_consensus.cc:374] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:24.987039  1490 raft_consensus.cc:729] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:24.987347  1490 consensus_queue.cc:260] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:24.987972  1490 ts_tablet_manager.cc:1398] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.001s
I20240503 02:11:24.990020  1493 tablet_bootstrap.cc:654] T d7f194bfa67c4aeea454483ecb13dcb3 P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:24.992050  1493 tablet_bootstrap.cc:492] T d7f194bfa67c4aeea454483ecb13dcb3 P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:24.992190  1493 ts_tablet_manager.cc:1367] T d7f194bfa67c4aeea454483ecb13dcb3 P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.008s	user 0.001s	sys 0.002s
I20240503 02:11:24.993126  1493 raft_consensus.cc:348] T d7f194bfa67c4aeea454483ecb13dcb3 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:24.993291  1493 raft_consensus.cc:374] T d7f194bfa67c4aeea454483ecb13dcb3 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:24.993399  1493 raft_consensus.cc:729] T d7f194bfa67c4aeea454483ecb13dcb3 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:24.993618  1493 consensus_queue.cc:260] T d7f194bfa67c4aeea454483ecb13dcb3 P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:24.994155  1493 ts_tablet_manager.cc:1398] T d7f194bfa67c4aeea454483ecb13dcb3 P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 02:11:24.995549  1498 raft_consensus.cc:348] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:24.996043  1498 raft_consensus.cc:374] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:24.996145  1498 raft_consensus.cc:729] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:24.996867  1498 consensus_queue.cc:260] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:24.998536  1498 ts_tablet_manager.cc:1398] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.004s
I20240503 02:11:24.998932  1498 tablet_bootstrap.cc:492] T d7f194bfa67c4aeea454483ecb13dcb3 P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:25.000314  1498 tablet_bootstrap.cc:654] T d7f194bfa67c4aeea454483ecb13dcb3 P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:25.002214  1498 tablet_bootstrap.cc:492] T d7f194bfa67c4aeea454483ecb13dcb3 P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:25.002362  1498 ts_tablet_manager.cc:1367] T d7f194bfa67c4aeea454483ecb13dcb3 P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:11:25.003185  1498 raft_consensus.cc:348] T d7f194bfa67c4aeea454483ecb13dcb3 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:25.003352  1498 raft_consensus.cc:374] T d7f194bfa67c4aeea454483ecb13dcb3 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:25.003451  1498 raft_consensus.cc:729] T d7f194bfa67c4aeea454483ecb13dcb3 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:25.003644  1498 consensus_queue.cc:260] T d7f194bfa67c4aeea454483ecb13dcb3 P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:25.004092  1498 ts_tablet_manager.cc:1398] T d7f194bfa67c4aeea454483ecb13dcb3 P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
W20240503 02:11:25.004467  1335 tablet.cc:2369] T d7f194bfa67c4aeea454483ecb13dcb3 P 572709bd9ae041a4a98029298a6f69c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:25.027144  1507 raft_consensus.cc:482] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:25.027428  1507 raft_consensus.cc:504] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:25.029644  1507 leader_election.cc:288] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
W20240503 02:11:25.031736  1204 tablet.cc:2369] T 16a92cef3eb143a5abcceffbd23cad67 P 288540692e7b404991570d9ca16ab6c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:25.038508  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb165523c1e849eeb8656e2fc23b31dd" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f" is_pre_election: true
I20240503 02:11:25.038508   896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb165523c1e849eeb8656e2fc23b31dd" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" is_pre_election: true
I20240503 02:11:25.039129   896 raft_consensus.cc:2454] T fb165523c1e849eeb8656e2fc23b31dd P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 0.
I20240503 02:11:25.039129  1027 raft_consensus.cc:2454] T fb165523c1e849eeb8656e2fc23b31dd P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 0.
I20240503 02:11:25.039899  1224 leader_election.cc:302] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:25.040355  1507 raft_consensus.cc:2781] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:25.040511  1507 raft_consensus.cc:482] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:25.040618  1507 raft_consensus.cc:3037] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:25.042735  1507 raft_consensus.cc:504] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:25.043396  1507 leader_election.cc:288] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 election: Requested vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:25.043980  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb165523c1e849eeb8656e2fc23b31dd" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f"
I20240503 02:11:25.043987   896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb165523c1e849eeb8656e2fc23b31dd" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
W20240503 02:11:25.044095  1073 tablet.cc:2369] T d7f194bfa67c4aeea454483ecb13dcb3 P f5e38dcb24084a1e969f83bf34aef88f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:25.044200  1027 raft_consensus.cc:3037] T fb165523c1e849eeb8656e2fc23b31dd P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:25.044210   896 raft_consensus.cc:3037] T fb165523c1e849eeb8656e2fc23b31dd P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:25.046923   896 raft_consensus.cc:2454] T fb165523c1e849eeb8656e2fc23b31dd P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 1.
I20240503 02:11:25.046932  1027 raft_consensus.cc:2454] T fb165523c1e849eeb8656e2fc23b31dd P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 1.
I20240503 02:11:25.047341  1223 leader_election.cc:302] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:25.047649  1507 raft_consensus.cc:2781] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:25.048064  1507 raft_consensus.cc:686] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [term 1 LEADER]: Becoming Leader. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Running, Role: LEADER
I20240503 02:11:25.048581  1507 consensus_queue.cc:237] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:25.054630   622 catalog_manager.cc:5484] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5). New cstate: current_term: 1 leader_uuid: "572709bd9ae041a4a98029298a6f69c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:25.058996   810 tablet.cc:2369] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:11:25.061154   942 tablet.cc:2369] T fb165523c1e849eeb8656e2fc23b31dd P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:25.128360  1502 raft_consensus.cc:482] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:25.128626  1502 raft_consensus.cc:504] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:25.130414  1502 leader_election.cc:288] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773), 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135)
I20240503 02:11:25.138221  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a92cef3eb143a5abcceffbd23cad67" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:25.138837  1158 raft_consensus.cc:2454] T 16a92cef3eb143a5abcceffbd23cad67 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 0.
I20240503 02:11:25.138945  1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a92cef3eb143a5abcceffbd23cad67" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e" is_pre_election: true
I20240503 02:11:25.139389  1420 raft_consensus.cc:2454] T 16a92cef3eb143a5abcceffbd23cad67 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 0.
I20240503 02:11:25.139488   699 leader_election.cc:302] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, e8d9a43983f2478ea7ababa09b230177; no voters: 
I20240503 02:11:25.139905  1502 raft_consensus.cc:2781] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:25.140046  1502 raft_consensus.cc:482] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:25.140148  1502 raft_consensus.cc:3037] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:25.142153  1502 raft_consensus.cc:504] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:25.142766  1502 leader_election.cc:288] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 election: Requested vote from peers 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773), 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135)
I20240503 02:11:25.143252  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a92cef3eb143a5abcceffbd23cad67" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:25.143321  1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a92cef3eb143a5abcceffbd23cad67" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e"
I20240503 02:11:25.143478  1158 raft_consensus.cc:3037] T 16a92cef3eb143a5abcceffbd23cad67 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:25.143528  1420 raft_consensus.cc:3037] T 16a92cef3eb143a5abcceffbd23cad67 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:25.147092  1158 raft_consensus.cc:2454] T 16a92cef3eb143a5abcceffbd23cad67 P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 1.
I20240503 02:11:25.147089  1420 raft_consensus.cc:2454] T 16a92cef3eb143a5abcceffbd23cad67 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 1.
I20240503 02:11:25.147565   699 leader_election.cc:302] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, e8d9a43983f2478ea7ababa09b230177; no voters: 
I20240503 02:11:25.147884  1502 raft_consensus.cc:2781] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:25.148694  1502 raft_consensus.cc:686] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [term 1 LEADER]: Becoming Leader. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Running, Role: LEADER
I20240503 02:11:25.149298  1502 consensus_queue.cc:237] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:25.154075   622 catalog_manager.cc:5484] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8d9a43983f2478ea7ababa09b230177 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "e8d9a43983f2478ea7ababa09b230177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:25.329969  1502 raft_consensus.cc:482] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:25.330212  1502 raft_consensus.cc:504] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:25.330931  1502 leader_election.cc:288] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:25.337538  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7f194bfa67c4aeea454483ecb13dcb3" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1" is_pre_election: true
I20240503 02:11:25.337721  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7f194bfa67c4aeea454483ecb13dcb3" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f" is_pre_election: true
I20240503 02:11:25.337945  1027 raft_consensus.cc:2454] T d7f194bfa67c4aeea454483ecb13dcb3 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 0.
I20240503 02:11:25.337965  1289 raft_consensus.cc:2454] T d7f194bfa67c4aeea454483ecb13dcb3 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 0.
I20240503 02:11:25.338435   698 leader_election.cc:302] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8d9a43983f2478ea7ababa09b230177, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:25.338802  1502 raft_consensus.cc:2781] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:25.338951  1502 raft_consensus.cc:482] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:25.339071  1502 raft_consensus.cc:3037] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:25.340919  1502 raft_consensus.cc:504] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:25.341444  1502 leader_election.cc:288] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 election: Requested vote from peers 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:25.341874  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7f194bfa67c4aeea454483ecb13dcb3" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f"
I20240503 02:11:25.341907  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7f194bfa67c4aeea454483ecb13dcb3" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1"
I20240503 02:11:25.342057  1289 raft_consensus.cc:3037] T d7f194bfa67c4aeea454483ecb13dcb3 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:25.342092  1027 raft_consensus.cc:3037] T d7f194bfa67c4aeea454483ecb13dcb3 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:25.344105  1027 raft_consensus.cc:2454] T d7f194bfa67c4aeea454483ecb13dcb3 P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 1.
I20240503 02:11:25.344125  1289 raft_consensus.cc:2454] T d7f194bfa67c4aeea454483ecb13dcb3 P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 1.
I20240503 02:11:25.344555   699 leader_election.cc:302] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, e8d9a43983f2478ea7ababa09b230177, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:25.344872  1502 raft_consensus.cc:2781] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:25.345057  1502 raft_consensus.cc:686] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [term 1 LEADER]: Becoming Leader. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Running, Role: LEADER
I20240503 02:11:25.345341  1502 consensus_queue.cc:237] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:25.348269   622 catalog_manager.cc:5484] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8d9a43983f2478ea7ababa09b230177 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "e8d9a43983f2478ea7ababa09b230177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:25.544667  1507 consensus_queue.cc:1035] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:25.570226  1507 consensus_queue.cc:1035] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:25.635730  1502 consensus_queue.cc:1035] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:25.646592  1515 consensus_queue.cc:1035] T 16a92cef3eb143a5abcceffbd23cad67 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:25.800489  1518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:25.800810  1518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:25.808737  1518 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 02:11:25.901196  1531 consensus_queue.cc:1035] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:25.902361  1531 consensus_queue.cc:1035] T d7f194bfa67c4aeea454483ecb13dcb3 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 361.649 ms
  time per row: 0.180824 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240503 02:11:27.273763   620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44878:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:27.283058   876 tablet_service.cc:1467] Processing CreateTablet for tablet 6947883c90ca46709a67ca185471178e (DEFAULT_TABLE table=rebalance_test_table_1 [id=4815754bb4b54261b8e6bd55234eccea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:27.283465   876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6947883c90ca46709a67ca185471178e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.284538   875 tablet_service.cc:1467] Processing CreateTablet for tablet 477c00ddd6e543e7b9da2f9a0561be51 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4815754bb4b54261b8e6bd55234eccea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:27.284734  1400 tablet_service.cc:1467] Processing CreateTablet for tablet 6947883c90ca46709a67ca185471178e (DEFAULT_TABLE table=rebalance_test_table_1 [id=4815754bb4b54261b8e6bd55234eccea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:27.284952   875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477c00ddd6e543e7b9da2f9a0561be51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.285166  1400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6947883c90ca46709a67ca185471178e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.285161  1269 tablet_service.cc:1467] Processing CreateTablet for tablet ca2c4b67a0fe426e9fe8a9b00360b12c (DEFAULT_TABLE table=rebalance_test_table_1 [id=4815754bb4b54261b8e6bd55234eccea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:27.285537  1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca2c4b67a0fe426e9fe8a9b00360b12c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.286084   744 tablet_service.cc:1467] Processing CreateTablet for tablet ca2c4b67a0fe426e9fe8a9b00360b12c (DEFAULT_TABLE table=rebalance_test_table_1 [id=4815754bb4b54261b8e6bd55234eccea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:27.286473   744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca2c4b67a0fe426e9fe8a9b00360b12c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.288286  1399 tablet_service.cc:1467] Processing CreateTablet for tablet 477c00ddd6e543e7b9da2f9a0561be51 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4815754bb4b54261b8e6bd55234eccea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:27.288657  1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477c00ddd6e543e7b9da2f9a0561be51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.291034  1138 tablet_service.cc:1467] Processing CreateTablet for tablet 6947883c90ca46709a67ca185471178e (DEFAULT_TABLE table=rebalance_test_table_1 [id=4815754bb4b54261b8e6bd55234eccea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:27.291440  1138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6947883c90ca46709a67ca185471178e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.292003  1600 tablet_bootstrap.cc:492] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:27.293658  1600 tablet_bootstrap.cc:654] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.294492  1007 tablet_service.cc:1467] Processing CreateTablet for tablet ca2c4b67a0fe426e9fe8a9b00360b12c (DEFAULT_TABLE table=rebalance_test_table_1 [id=4815754bb4b54261b8e6bd55234eccea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:27.295045  1007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca2c4b67a0fe426e9fe8a9b00360b12c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.295123  1137 tablet_service.cc:1467] Processing CreateTablet for tablet 477c00ddd6e543e7b9da2f9a0561be51 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4815754bb4b54261b8e6bd55234eccea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:27.295603  1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 477c00ddd6e543e7b9da2f9a0561be51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:27.300985  1599 tablet_bootstrap.cc:492] T 6947883c90ca46709a67ca185471178e P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:27.302979  1599 tablet_bootstrap.cc:654] T 6947883c90ca46709a67ca185471178e P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.323523  1604 tablet_bootstrap.cc:492] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:27.324259  1606 tablet_bootstrap.cc:492] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:27.325603  1604 tablet_bootstrap.cc:654] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.326402  1603 tablet_bootstrap.cc:492] T ca2c4b67a0fe426e9fe8a9b00360b12c P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:27.326922  1605 tablet_bootstrap.cc:492] T ca2c4b67a0fe426e9fe8a9b00360b12c P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:27.327912  1606 tablet_bootstrap.cc:654] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.328240  1603 tablet_bootstrap.cc:654] T ca2c4b67a0fe426e9fe8a9b00360b12c P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.331509  1605 tablet_bootstrap.cc:654] T ca2c4b67a0fe426e9fe8a9b00360b12c P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.334103  1599 tablet_bootstrap.cc:492] T 6947883c90ca46709a67ca185471178e P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:27.334327  1599 ts_tablet_manager.cc:1367] T 6947883c90ca46709a67ca185471178e P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.033s	user 0.009s	sys 0.018s
I20240503 02:11:27.334753  1600 tablet_bootstrap.cc:492] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:27.334820  1605 tablet_bootstrap.cc:492] T ca2c4b67a0fe426e9fe8a9b00360b12c P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:27.334836  1604 tablet_bootstrap.cc:492] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:27.334949  1600 ts_tablet_manager.cc:1367] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.043s	user 0.021s	sys 0.021s
I20240503 02:11:27.334982  1604 ts_tablet_manager.cc:1367] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.000s
I20240503 02:11:27.335022  1605 ts_tablet_manager.cc:1367] T ca2c4b67a0fe426e9fe8a9b00360b12c P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20240503 02:11:27.335650  1599 raft_consensus.cc:348] T 6947883c90ca46709a67ca185471178e P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:27.335896  1603 tablet_bootstrap.cc:492] T ca2c4b67a0fe426e9fe8a9b00360b12c P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:27.335819  1605 raft_consensus.cc:348] T ca2c4b67a0fe426e9fe8a9b00360b12c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:27.335999  1599 raft_consensus.cc:374] T 6947883c90ca46709a67ca185471178e P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.335878  1600 raft_consensus.cc:348] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:27.336051  1603 ts_tablet_manager.cc:1367] T ca2c4b67a0fe426e9fe8a9b00360b12c P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.002s
I20240503 02:11:27.336087  1600 raft_consensus.cc:374] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.336103  1605 raft_consensus.cc:374] T ca2c4b67a0fe426e9fe8a9b00360b12c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.336166  1600 raft_consensus.cc:729] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.336212  1599 raft_consensus.cc:729] T 6947883c90ca46709a67ca185471178e P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.336283  1605 raft_consensus.cc:729] T ca2c4b67a0fe426e9fe8a9b00360b12c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.336457  1600 consensus_queue.cc:260] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:27.336578  1599 consensus_queue.cc:260] T 6947883c90ca46709a67ca185471178e P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:27.336603  1605 consensus_queue.cc:260] T ca2c4b67a0fe426e9fe8a9b00360b12c P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:27.337956  1606 tablet_bootstrap.cc:492] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:27.338203  1606 ts_tablet_manager.cc:1367] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.000s
I20240503 02:11:27.339308  1604 raft_consensus.cc:348] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:27.339522  1604 raft_consensus.cc:374] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.339259  1606 raft_consensus.cc:348] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:27.339622  1604 raft_consensus.cc:729] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.339754  1606 raft_consensus.cc:374] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.339917  1605 ts_tablet_manager.cc:1398] T ca2c4b67a0fe426e9fe8a9b00360b12c P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:11:27.339915  1606 raft_consensus.cc:729] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.339807  1603 raft_consensus.cc:348] T ca2c4b67a0fe426e9fe8a9b00360b12c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:27.340044  1603 raft_consensus.cc:374] T ca2c4b67a0fe426e9fe8a9b00360b12c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.340140  1603 raft_consensus.cc:729] T ca2c4b67a0fe426e9fe8a9b00360b12c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.340268  1606 consensus_queue.cc:260] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:27.340397  1603 consensus_queue.cc:260] T ca2c4b67a0fe426e9fe8a9b00360b12c P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:27.340929  1603 ts_tablet_manager.cc:1398] T ca2c4b67a0fe426e9fe8a9b00360b12c P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240503 02:11:27.341208  1604 consensus_queue.cc:260] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:27.344993  1604 ts_tablet_manager.cc:1398] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20240503 02:11:27.346395  1606 ts_tablet_manager.cc:1398] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I20240503 02:11:27.347416  1604 tablet_bootstrap.cc:492] T 477c00ddd6e543e7b9da2f9a0561be51 P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:27.350524  1604 tablet_bootstrap.cc:654] T 477c00ddd6e543e7b9da2f9a0561be51 P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.352252  1599 ts_tablet_manager.cc:1398] T 6947883c90ca46709a67ca185471178e P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.018s	user 0.000s	sys 0.008s
I20240503 02:11:27.352669  1599 tablet_bootstrap.cc:492] T 477c00ddd6e543e7b9da2f9a0561be51 P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:27.352679  1600 ts_tablet_manager.cc:1398] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.006s
I20240503 02:11:27.353158  1600 tablet_bootstrap.cc:492] T 6947883c90ca46709a67ca185471178e P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:27.354372  1599 tablet_bootstrap.cc:654] T 477c00ddd6e543e7b9da2f9a0561be51 P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.354897  1600 tablet_bootstrap.cc:654] T 6947883c90ca46709a67ca185471178e P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:27.355827  1604 tablet_bootstrap.cc:492] T 477c00ddd6e543e7b9da2f9a0561be51 P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:27.356012  1604 ts_tablet_manager.cc:1367] T 477c00ddd6e543e7b9da2f9a0561be51 P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.002s
I20240503 02:11:27.356796  1604 raft_consensus.cc:348] T 477c00ddd6e543e7b9da2f9a0561be51 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:27.357568  1604 raft_consensus.cc:374] T 477c00ddd6e543e7b9da2f9a0561be51 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.357697  1604 raft_consensus.cc:729] T 477c00ddd6e543e7b9da2f9a0561be51 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.358032  1604 consensus_queue.cc:260] T 477c00ddd6e543e7b9da2f9a0561be51 P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:27.359527  1604 ts_tablet_manager.cc:1398] T 477c00ddd6e543e7b9da2f9a0561be51 P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 02:11:27.359795  1600 tablet_bootstrap.cc:492] T 6947883c90ca46709a67ca185471178e P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:27.359980  1600 ts_tablet_manager.cc:1367] T 6947883c90ca46709a67ca185471178e P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.003s
I20240503 02:11:27.360081  1599 tablet_bootstrap.cc:492] T 477c00ddd6e543e7b9da2f9a0561be51 P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:27.360255  1599 ts_tablet_manager.cc:1367] T 477c00ddd6e543e7b9da2f9a0561be51 P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.002s
I20240503 02:11:27.360716  1600 raft_consensus.cc:348] T 6947883c90ca46709a67ca185471178e P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:27.360956  1600 raft_consensus.cc:374] T 6947883c90ca46709a67ca185471178e P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.361075  1600 raft_consensus.cc:729] T 6947883c90ca46709a67ca185471178e P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.361059  1599 raft_consensus.cc:348] T 477c00ddd6e543e7b9da2f9a0561be51 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:27.361301  1599 raft_consensus.cc:374] T 477c00ddd6e543e7b9da2f9a0561be51 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:27.361430  1599 raft_consensus.cc:729] T 477c00ddd6e543e7b9da2f9a0561be51 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:27.361474  1600 consensus_queue.cc:260] T 6947883c90ca46709a67ca185471178e P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:27.361692  1599 consensus_queue.cc:260] T 477c00ddd6e543e7b9da2f9a0561be51 P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:27.362663  1600 ts_tablet_manager.cc:1398] T 6947883c90ca46709a67ca185471178e P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.001s
I20240503 02:11:27.362860  1599 ts_tablet_manager.cc:1398] T 477c00ddd6e543e7b9da2f9a0561be51 P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:11:27.391673  1611 raft_consensus.cc:482] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:27.391932  1611 raft_consensus.cc:504] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:27.393581  1611 leader_election.cc:288] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:27.401906  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477c00ddd6e543e7b9da2f9a0561be51" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:27.401998  1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477c00ddd6e543e7b9da2f9a0561be51" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e" is_pre_election: true
I20240503 02:11:27.402173  1158 raft_consensus.cc:2454] T 477c00ddd6e543e7b9da2f9a0561be51 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 0.
I20240503 02:11:27.402236  1420 raft_consensus.cc:2454] T 477c00ddd6e543e7b9da2f9a0561be51 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 0.
I20240503 02:11:27.402691   831 leader_election.cc:302] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:27.403071  1611 raft_consensus.cc:2781] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:27.403182  1611 raft_consensus.cc:482] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:27.403293  1611 raft_consensus.cc:3037] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.405257  1611 raft_consensus.cc:504] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:27.405840  1611 leader_election.cc:288] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 election: Requested vote from peers 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:27.406236  1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477c00ddd6e543e7b9da2f9a0561be51" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e"
I20240503 02:11:27.406311  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477c00ddd6e543e7b9da2f9a0561be51" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:27.406437  1420 raft_consensus.cc:3037] T 477c00ddd6e543e7b9da2f9a0561be51 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.406544  1158 raft_consensus.cc:3037] T 477c00ddd6e543e7b9da2f9a0561be51 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.408288  1420 raft_consensus.cc:2454] T 477c00ddd6e543e7b9da2f9a0561be51 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 1.
I20240503 02:11:27.408430  1158 raft_consensus.cc:2454] T 477c00ddd6e543e7b9da2f9a0561be51 P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 1.
I20240503 02:11:27.408694   831 leader_election.cc:302] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:27.409006  1611 raft_consensus.cc:2781] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:27.409425  1611 raft_consensus.cc:686] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 LEADER]: Becoming Leader. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Running, Role: LEADER
I20240503 02:11:27.409960  1611 consensus_queue.cc:237] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:27.413300   621 catalog_manager.cc:5484] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2). New cstate: current_term: 1 leader_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:27.416622  1614 raft_consensus.cc:482] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:27.416795  1614 raft_consensus.cc:504] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:27.418370  1614 leader_election.cc:288] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135)
I20240503 02:11:27.425487  1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6947883c90ca46709a67ca185471178e" candidate_uuid: "288540692e7b404991570d9ca16ab6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e" is_pre_election: true
I20240503 02:11:27.425743  1420 raft_consensus.cc:2454] T 6947883c90ca46709a67ca185471178e P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 288540692e7b404991570d9ca16ab6c0 in term 0.
I20240503 02:11:27.425698   896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6947883c90ca46709a67ca185471178e" candidate_uuid: "288540692e7b404991570d9ca16ab6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" is_pre_election: true
I20240503 02:11:27.425966   896 raft_consensus.cc:2454] T 6947883c90ca46709a67ca185471178e P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 288540692e7b404991570d9ca16ab6c0 in term 0.
I20240503 02:11:27.426276  1093 leader_election.cc:302] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 4fda3b525c234f8f88aaefd355e7274e; no voters: 
I20240503 02:11:27.426729  1614 raft_consensus.cc:2781] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:27.426848  1614 raft_consensus.cc:482] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:27.426957  1614 raft_consensus.cc:3037] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.428931  1614 raft_consensus.cc:504] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:27.429522  1614 leader_election.cc:288] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [CANDIDATE]: Term 1 election: Requested vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135)
I20240503 02:11:27.429977   896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6947883c90ca46709a67ca185471178e" candidate_uuid: "288540692e7b404991570d9ca16ab6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
I20240503 02:11:27.430078  1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6947883c90ca46709a67ca185471178e" candidate_uuid: "288540692e7b404991570d9ca16ab6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e"
I20240503 02:11:27.430220   896 raft_consensus.cc:3037] T 6947883c90ca46709a67ca185471178e P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.430312  1420 raft_consensus.cc:3037] T 6947883c90ca46709a67ca185471178e P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.433193   896 raft_consensus.cc:2454] T 6947883c90ca46709a67ca185471178e P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 288540692e7b404991570d9ca16ab6c0 in term 1.
I20240503 02:11:27.433207  1420 raft_consensus.cc:2454] T 6947883c90ca46709a67ca185471178e P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 288540692e7b404991570d9ca16ab6c0 in term 1.
I20240503 02:11:27.433720  1093 leader_election.cc:302] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:27.433912  1613 raft_consensus.cc:482] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:27.434096  1614 raft_consensus.cc:2781] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:27.434101  1613 raft_consensus.cc:504] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:27.434588  1614 raft_consensus.cc:686] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [term 1 LEADER]: Becoming Leader. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Running, Role: LEADER
I20240503 02:11:27.435130  1614 consensus_queue.cc:237] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:27.435817  1613 leader_election.cc:288] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189), e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401)
I20240503 02:11:27.460589   621 catalog_manager.cc:5484] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 288540692e7b404991570d9ca16ab6c0 (127.0.145.4). New cstate: current_term: 1 leader_uuid: "288540692e7b404991570d9ca16ab6c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:27.464313   764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2c4b67a0fe426e9fe8a9b00360b12c" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177" is_pre_election: true
I20240503 02:11:27.464635   764 raft_consensus.cc:2454] T ca2c4b67a0fe426e9fe8a9b00360b12c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 0.
I20240503 02:11:27.464854  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2c4b67a0fe426e9fe8a9b00360b12c" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1" is_pre_election: true
I20240503 02:11:27.465097  1289 raft_consensus.cc:2454] T ca2c4b67a0fe426e9fe8a9b00360b12c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 0.
I20240503 02:11:27.465201   961 leader_election.cc:302] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8d9a43983f2478ea7ababa09b230177, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:27.465615  1613 raft_consensus.cc:2781] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:27.465744  1613 raft_consensus.cc:482] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:27.465850  1613 raft_consensus.cc:3037] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.467805  1613 raft_consensus.cc:504] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:27.468343  1613 leader_election.cc:288] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 election: Requested vote from peers 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189), e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401)
I20240503 02:11:27.468775  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2c4b67a0fe426e9fe8a9b00360b12c" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1"
I20240503 02:11:27.468816   764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca2c4b67a0fe426e9fe8a9b00360b12c" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177"
I20240503 02:11:27.468981  1289 raft_consensus.cc:3037] T ca2c4b67a0fe426e9fe8a9b00360b12c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.468993   764 raft_consensus.cc:3037] T ca2c4b67a0fe426e9fe8a9b00360b12c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:27.470836  1289 raft_consensus.cc:2454] T ca2c4b67a0fe426e9fe8a9b00360b12c P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 1.
I20240503 02:11:27.470856   764 raft_consensus.cc:2454] T ca2c4b67a0fe426e9fe8a9b00360b12c P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 1.
I20240503 02:11:27.471240   961 leader_election.cc:302] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8d9a43983f2478ea7ababa09b230177, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:27.471546  1613 raft_consensus.cc:2781] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:27.471946  1613 raft_consensus.cc:686] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [term 1 LEADER]: Becoming Leader. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Running, Role: LEADER
I20240503 02:11:27.472411  1613 consensus_queue.cc:237] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:27.475849   621 catalog_manager.cc:5484] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f reported cstate change: term changed from 0 to 1, leader changed from <none> to f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3). New cstate: current_term: 1 leader_uuid: "f5e38dcb24084a1e969f83bf34aef88f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:27.828048  1611 consensus_queue.cc:1035] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:27.828820  1635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:27.829157  1635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:11:27.834420  1611 consensus_queue.cc:1035] T 477c00ddd6e543e7b9da2f9a0561be51 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:27.840708  1635 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:11:27.858171  1614 consensus_queue.cc:1035] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:27.864220  1627 consensus_queue.cc:1035] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240503 02:11:28.052973  1289 raft_consensus.cc:1261] T ca2c4b67a0fe426e9fe8a9b00360b12c P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Refusing update from remote peer f5e38dcb24084a1e969f83bf34aef88f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:28.060485  1677 consensus_queue.cc:1035] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.100035   764 raft_consensus.cc:1261] T ca2c4b67a0fe426e9fe8a9b00360b12c P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Refusing update from remote peer f5e38dcb24084a1e969f83bf34aef88f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:11:28.101444  1678 consensus_queue.cc:1035] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.117142  1672 mvcc.cc:205] Tried to move back new op lower bound from 7023420571637420032 to 7023420569490579456. Current Snapshot: MvccSnapshot[applied={T|T < 7023420571637420032}]
I20240503 02:11:28.166851  1687 mvcc.cc:205] Tried to move back new op lower bound from 7023420571637420032 to 7023420569490579456. Current Snapshot: MvccSnapshot[applied={T|T < 7023420571637420032}]
I20240503 02:11:28.175634  1695 mvcc.cc:205] Tried to move back new op lower bound from 7023420571637420032 to 7023420569490579456. Current Snapshot: MvccSnapshot[applied={T|T < 7023420571637420032}]

INSERT report
    rows total: 2000
    time total: 306.004 ms
  time per row: 0.153002 ms
I20240503 02:11:28.326242   622 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44878:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:28.334731  1007 tablet_service.cc:1467] Processing CreateTablet for tablet 18ccfc8dbf884ab3a3e3338d679787af (DEFAULT_TABLE table=rebalance_test_table_2 [id=4965a6a71b9842259d49dd61d8be11fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:28.334937  1269 tablet_service.cc:1467] Processing CreateTablet for tablet 18ccfc8dbf884ab3a3e3338d679787af (DEFAULT_TABLE table=rebalance_test_table_2 [id=4965a6a71b9842259d49dd61d8be11fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:28.335245  1007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18ccfc8dbf884ab3a3e3338d679787af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.335366  1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18ccfc8dbf884ab3a3e3338d679787af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.335433  1138 tablet_service.cc:1467] Processing CreateTablet for tablet c218fde67d2b41df8bd27d7cf4e1fe38 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4965a6a71b9842259d49dd61d8be11fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:28.335842  1138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c218fde67d2b41df8bd27d7cf4e1fe38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.335886   744 tablet_service.cc:1467] Processing CreateTablet for tablet 18ccfc8dbf884ab3a3e3338d679787af (DEFAULT_TABLE table=rebalance_test_table_2 [id=4965a6a71b9842259d49dd61d8be11fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:28.336277   744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18ccfc8dbf884ab3a3e3338d679787af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.341486  1399 tablet_service.cc:1467] Processing CreateTablet for tablet c218fde67d2b41df8bd27d7cf4e1fe38 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4965a6a71b9842259d49dd61d8be11fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:28.342082  1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c218fde67d2b41df8bd27d7cf4e1fe38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.341964   743 tablet_service.cc:1467] Processing CreateTablet for tablet 90223cd1477f46788581d2dcc937e162 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4965a6a71b9842259d49dd61d8be11fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:28.342509   743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90223cd1477f46788581d2dcc937e162. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.344055   876 tablet_service.cc:1467] Processing CreateTablet for tablet c218fde67d2b41df8bd27d7cf4e1fe38 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4965a6a71b9842259d49dd61d8be11fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:28.344640   876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c218fde67d2b41df8bd27d7cf4e1fe38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.350467  1709 tablet_bootstrap.cc:492] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:28.352543  1709 tablet_bootstrap.cc:654] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:28.353332  1269 tablet_service.cc:1467] Processing CreateTablet for tablet 90223cd1477f46788581d2dcc937e162 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4965a6a71b9842259d49dd61d8be11fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:28.353911  1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90223cd1477f46788581d2dcc937e162. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.353818  1006 tablet_service.cc:1467] Processing CreateTablet for tablet 90223cd1477f46788581d2dcc937e162 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4965a6a71b9842259d49dd61d8be11fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:28.354318  1006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90223cd1477f46788581d2dcc937e162. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:28.365351  1713 tablet_bootstrap.cc:492] T c218fde67d2b41df8bd27d7cf4e1fe38 P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:28.367290  1713 tablet_bootstrap.cc:654] T c218fde67d2b41df8bd27d7cf4e1fe38 P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:28.371568  1709 tablet_bootstrap.cc:492] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:28.371757  1709 ts_tablet_manager.cc:1367] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.021s	user 0.005s	sys 0.010s
I20240503 02:11:28.372736  1709 raft_consensus.cc:348] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:28.372985  1709 raft_consensus.cc:374] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:28.374006  1709 raft_consensus.cc:729] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:28.374284  1709 consensus_queue.cc:260] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:28.375036  1709 ts_tablet_manager.cc:1398] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:28.376763  1709 tablet_bootstrap.cc:492] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:28.376777  1712 tablet_bootstrap.cc:492] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:28.378010  1711 tablet_bootstrap.cc:492] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:28.378700  1709 tablet_bootstrap.cc:654] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:28.380249  1711 tablet_bootstrap.cc:654] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:28.382365  1712 tablet_bootstrap.cc:654] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:28.382649  1708 tablet_bootstrap.cc:492] T 18ccfc8dbf884ab3a3e3338d679787af P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:28.382820  1715 tablet_bootstrap.cc:492] T 18ccfc8dbf884ab3a3e3338d679787af P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:28.385016  1708 tablet_bootstrap.cc:654] T 18ccfc8dbf884ab3a3e3338d679787af P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:28.385149  1715 tablet_bootstrap.cc:654] T 18ccfc8dbf884ab3a3e3338d679787af P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:28.388471  1711 tablet_bootstrap.cc:492] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:28.388782  1711 ts_tablet_manager.cc:1367] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.000s
I20240503 02:11:28.389971  1711 raft_consensus.cc:348] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:28.390364  1711 raft_consensus.cc:374] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:28.390538  1711 raft_consensus.cc:729] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:28.391113  1709 tablet_bootstrap.cc:492] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:28.390983  1711 consensus_queue.cc:260] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:28.391341  1709 ts_tablet_manager.cc:1367] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.015s	user 0.002s	sys 0.004s
I20240503 02:11:28.391912  1711 ts_tablet_manager.cc:1398] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:28.392525  1709 raft_consensus.cc:348] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:28.392833  1709 raft_consensus.cc:374] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:28.392961  1709 raft_consensus.cc:729] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:28.393270  1709 consensus_queue.cc:260] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:28.393905  1709 ts_tablet_manager.cc:1398] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:28.396060  1713 tablet_bootstrap.cc:492] T c218fde67d2b41df8bd27d7cf4e1fe38 P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:28.396292  1713 ts_tablet_manager.cc:1367] T c218fde67d2b41df8bd27d7cf4e1fe38 P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.031s	user 0.006s	sys 0.006s
I20240503 02:11:28.396533  1712 tablet_bootstrap.cc:492] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:28.396800  1712 ts_tablet_manager.cc:1367] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.020s	user 0.005s	sys 0.000s
I20240503 02:11:28.397223  1713 raft_consensus.cc:348] T c218fde67d2b41df8bd27d7cf4e1fe38 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:28.397481  1713 raft_consensus.cc:374] T c218fde67d2b41df8bd27d7cf4e1fe38 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:28.397617  1713 raft_consensus.cc:729] T c218fde67d2b41df8bd27d7cf4e1fe38 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:28.397917  1713 consensus_queue.cc:260] T c218fde67d2b41df8bd27d7cf4e1fe38 P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:28.398245  1712 raft_consensus.cc:348] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:28.398581  1712 raft_consensus.cc:374] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:28.398773  1712 raft_consensus.cc:729] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:28.399117  1713 ts_tablet_manager.cc:1398] T c218fde67d2b41df8bd27d7cf4e1fe38 P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 02:11:28.399713  1712 consensus_queue.cc:260] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:28.400666  1715 tablet_bootstrap.cc:492] T 18ccfc8dbf884ab3a3e3338d679787af P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:28.400703  1708 tablet_bootstrap.cc:492] T 18ccfc8dbf884ab3a3e3338d679787af P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:28.400833  1715 ts_tablet_manager.cc:1367] T 18ccfc8dbf884ab3a3e3338d679787af P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.018s	user 0.006s	sys 0.003s
I20240503 02:11:28.400918  1708 ts_tablet_manager.cc:1367] T 18ccfc8dbf884ab3a3e3338d679787af P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.018s	user 0.003s	sys 0.006s
I20240503 02:11:28.401609  1715 raft_consensus.cc:348] T 18ccfc8dbf884ab3a3e3338d679787af P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:28.401880  1715 raft_consensus.cc:374] T 18ccfc8dbf884ab3a3e3338d679787af P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:28.402010  1715 raft_consensus.cc:729] T 18ccfc8dbf884ab3a3e3338d679787af P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:28.401876  1708 raft_consensus.cc:348] T 18ccfc8dbf884ab3a3e3338d679787af P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:28.402210  1708 raft_consensus.cc:374] T 18ccfc8dbf884ab3a3e3338d679787af P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:28.402360  1708 raft_consensus.cc:729] T 18ccfc8dbf884ab3a3e3338d679787af P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:28.402329  1715 consensus_queue.cc:260] T 18ccfc8dbf884ab3a3e3338d679787af P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:28.402765  1708 consensus_queue.cc:260] T 18ccfc8dbf884ab3a3e3338d679787af P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:28.403920  1715 ts_tablet_manager.cc:1398] T 18ccfc8dbf884ab3a3e3338d679787af P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 02:11:28.404179  1712 ts_tablet_manager.cc:1398] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.003s
I20240503 02:11:28.404377  1715 tablet_bootstrap.cc:492] T 90223cd1477f46788581d2dcc937e162 P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:28.403434  1708 ts_tablet_manager.cc:1398] T 18ccfc8dbf884ab3a3e3338d679787af P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:28.404661  1708 tablet_bootstrap.cc:492] T 90223cd1477f46788581d2dcc937e162 P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:28.406210  1715 tablet_bootstrap.cc:654] T 90223cd1477f46788581d2dcc937e162 P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:28.406299  1708 tablet_bootstrap.cc:654] T 90223cd1477f46788581d2dcc937e162 P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:28.411769  1715 tablet_bootstrap.cc:492] T 90223cd1477f46788581d2dcc937e162 P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:28.411839  1708 tablet_bootstrap.cc:492] T 90223cd1477f46788581d2dcc937e162 P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:28.411962  1715 ts_tablet_manager.cc:1367] T 90223cd1477f46788581d2dcc937e162 P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.003s
I20240503 02:11:28.412000  1708 ts_tablet_manager.cc:1367] T 90223cd1477f46788581d2dcc937e162 P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240503 02:11:28.412678  1708 raft_consensus.cc:348] T 90223cd1477f46788581d2dcc937e162 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:28.412707  1715 raft_consensus.cc:348] T 90223cd1477f46788581d2dcc937e162 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:28.412940  1715 raft_consensus.cc:374] T 90223cd1477f46788581d2dcc937e162 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:28.412951  1708 raft_consensus.cc:374] T 90223cd1477f46788581d2dcc937e162 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:28.413029  1708 raft_consensus.cc:729] T 90223cd1477f46788581d2dcc937e162 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:28.413031  1715 raft_consensus.cc:729] T 90223cd1477f46788581d2dcc937e162 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:28.413275  1708 consensus_queue.cc:260] T 90223cd1477f46788581d2dcc937e162 P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:28.413304  1715 consensus_queue.cc:260] T 90223cd1477f46788581d2dcc937e162 P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:28.413781  1715 ts_tablet_manager.cc:1398] T 90223cd1477f46788581d2dcc937e162 P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:11:28.414305  1708 ts_tablet_manager.cc:1398] T 90223cd1477f46788581d2dcc937e162 P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 02:11:28.428788  1716 raft_consensus.cc:482] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:28.429013  1716 raft_consensus.cc:504] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:28.429785  1716 leader_election.cc:288] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:28.430979  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90223cd1477f46788581d2dcc937e162" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f" is_pre_election: true
I20240503 02:11:28.431221  1027 raft_consensus.cc:2454] T 90223cd1477f46788581d2dcc937e162 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 0.
I20240503 02:11:28.431697  1223 leader_election.cc:302] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:28.433027  1716 raft_consensus.cc:2781] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:28.433127  1716 raft_consensus.cc:482] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:28.433212  1716 raft_consensus.cc:3037] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:28.436206  1716 raft_consensus.cc:504] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:28.436808  1716 leader_election.cc:288] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 election: Requested vote from peers e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:28.437013   764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90223cd1477f46788581d2dcc937e162" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177" is_pre_election: true
I20240503 02:11:28.437126   763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90223cd1477f46788581d2dcc937e162" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177"
I20240503 02:11:28.437283   764 raft_consensus.cc:2454] T 90223cd1477f46788581d2dcc937e162 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 0.
I20240503 02:11:28.437299  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90223cd1477f46788581d2dcc937e162" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f"
I20240503 02:11:28.437466  1027 raft_consensus.cc:3037] T 90223cd1477f46788581d2dcc937e162 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:28.439400  1027 raft_consensus.cc:2454] T 90223cd1477f46788581d2dcc937e162 P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 1.
I20240503 02:11:28.439756  1223 leader_election.cc:302] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, f5e38dcb24084a1e969f83bf34aef88f; no voters: e8d9a43983f2478ea7ababa09b230177
I20240503 02:11:28.440076  1716 raft_consensus.cc:2781] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:28.440197  1716 raft_consensus.cc:686] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [term 1 LEADER]: Becoming Leader. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Running, Role: LEADER
I20240503 02:11:28.440477  1716 consensus_queue.cc:237] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:28.443135   620 catalog_manager.cc:5484] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5). New cstate: current_term: 1 leader_uuid: "572709bd9ae041a4a98029298a6f69c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:28.570056  1716 raft_consensus.cc:482] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:28.570266  1716 raft_consensus.cc:504] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:28.570880  1716 leader_election.cc:288] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511), e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401)
I20240503 02:11:28.571475  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ccfc8dbf884ab3a3e3338d679787af" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f" is_pre_election: true
I20240503 02:11:28.571543   764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ccfc8dbf884ab3a3e3338d679787af" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177" is_pre_election: true
I20240503 02:11:28.571681  1027 raft_consensus.cc:2454] T 18ccfc8dbf884ab3a3e3338d679787af P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 0.
I20240503 02:11:28.571774   764 raft_consensus.cc:2454] T 18ccfc8dbf884ab3a3e3338d679787af P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 0.
I20240503 02:11:28.572088  1223 leader_election.cc:302] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:28.572348  1716 raft_consensus.cc:2781] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:28.572450  1716 raft_consensus.cc:482] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:28.572549  1716 raft_consensus.cc:3037] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:28.574446  1716 raft_consensus.cc:504] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:28.575019  1716 leader_election.cc:288] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 election: Requested vote from peers f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511), e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401)
I20240503 02:11:28.575407  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ccfc8dbf884ab3a3e3338d679787af" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f"
I20240503 02:11:28.575465   764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ccfc8dbf884ab3a3e3338d679787af" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177"
I20240503 02:11:28.575582  1027 raft_consensus.cc:3037] T 18ccfc8dbf884ab3a3e3338d679787af P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:28.575656   764 raft_consensus.cc:3037] T 18ccfc8dbf884ab3a3e3338d679787af P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:28.577366  1027 raft_consensus.cc:2454] T 18ccfc8dbf884ab3a3e3338d679787af P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 1.
I20240503 02:11:28.577600   764 raft_consensus.cc:2454] T 18ccfc8dbf884ab3a3e3338d679787af P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 1.
I20240503 02:11:28.577802  1223 leader_election.cc:302] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:28.578069  1716 raft_consensus.cc:2781] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:28.578243  1716 raft_consensus.cc:686] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [term 1 LEADER]: Becoming Leader. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Running, Role: LEADER
I20240503 02:11:28.578502  1716 consensus_queue.cc:237] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:28.581224   620 catalog_manager.cc:5484] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5). New cstate: current_term: 1 leader_uuid: "572709bd9ae041a4a98029298a6f69c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:28.823944  1722 raft_consensus.cc:482] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:28.824177  1722 raft_consensus.cc:504] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:28.825609  1722 leader_election.cc:288] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773), 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485)
I20240503 02:11:28.827672  1611 raft_consensus.cc:482] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:28.827873  1611 raft_consensus.cc:504] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:28.828600  1611 leader_election.cc:288] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773), 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135)
I20240503 02:11:28.829114  1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c218fde67d2b41df8bd27d7cf4e1fe38" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:28.829114  1419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c218fde67d2b41df8bd27d7cf4e1fe38" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e" is_pre_election: true
I20240503 02:11:28.829334  1419 raft_consensus.cc:2454] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 0.
I20240503 02:11:28.829353  1157 raft_consensus.cc:2454] T c218fde67d2b41df8bd27d7cf4e1fe38 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 0.
I20240503 02:11:28.829802   831 leader_election.cc:302] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:28.830101  1611 raft_consensus.cc:2781] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:28.830200  1611 raft_consensus.cc:482] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:28.830317  1611 raft_consensus.cc:3037] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:28.832684  1611 raft_consensus.cc:504] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:28.833621  1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c218fde67d2b41df8bd27d7cf4e1fe38" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:28.833658   896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c218fde67d2b41df8bd27d7cf4e1fe38" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" is_pre_election: true
I20240503 02:11:28.833670  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c218fde67d2b41df8bd27d7cf4e1fe38" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:28.833817  1157 raft_consensus.cc:2454] T c218fde67d2b41df8bd27d7cf4e1fe38 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:28.833931   896 raft_consensus.cc:2379] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4fda3b525c234f8f88aaefd355e7274e in current term 1: Already voted for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in this term.
I20240503 02:11:28.834290  1611 leader_election.cc:288] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 election: Requested vote from peers 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773), 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135)
I20240503 02:11:28.834311  1355 leader_election.cc:302] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 4fda3b525c234f8f88aaefd355e7274e; no voters: 
I20240503 02:11:28.834565  1419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c218fde67d2b41df8bd27d7cf4e1fe38" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e"
I20240503 02:11:28.834784  1722 raft_consensus.cc:2781] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:28.834942  1722 raft_consensus.cc:482] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:28.835041  1722 raft_consensus.cc:3037] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:28.837021  1722 raft_consensus.cc:504] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:28.837433  1419 raft_consensus.cc:2379] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in current term 1: Already voted for candidate 4fda3b525c234f8f88aaefd355e7274e in this term.
I20240503 02:11:28.837730  1722 leader_election.cc:288] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Requested vote from peers 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773), 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485)
I20240503 02:11:28.838088  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c218fde67d2b41df8bd27d7cf4e1fe38" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:28.838294  1158 raft_consensus.cc:3037] T c218fde67d2b41df8bd27d7cf4e1fe38 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:28.838268   896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c218fde67d2b41df8bd27d7cf4e1fe38" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
I20240503 02:11:28.838266   831 leader_election.cc:302] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 288540692e7b404991570d9ca16ab6c0, 4fda3b525c234f8f88aaefd355e7274e
I20240503 02:11:28.838518   896 raft_consensus.cc:2379] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fda3b525c234f8f88aaefd355e7274e in current term 1: Already voted for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in this term.
I20240503 02:11:28.838706  1611 raft_consensus.cc:2726] T c218fde67d2b41df8bd27d7cf4e1fe38 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:11:28.840189  1158 raft_consensus.cc:2454] T c218fde67d2b41df8bd27d7cf4e1fe38 P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:28.840528  1355 leader_election.cc:302] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 4fda3b525c234f8f88aaefd355e7274e; no voters: 88791c3c9c9341a6a2d8a6b5d3aa1f1b
I20240503 02:11:28.840822  1722 raft_consensus.cc:2781] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:28.841166  1722 raft_consensus.cc:686] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [term 1 LEADER]: Becoming Leader. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Running, Role: LEADER
I20240503 02:11:28.841807  1722 consensus_queue.cc:237] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:28.843665  1716 consensus_queue.cc:1035] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.845491   620 catalog_manager.cc:5484] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6). New cstate: current_term: 1 leader_uuid: "4fda3b525c234f8f88aaefd355e7274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:28.851490   764 raft_consensus.cc:3037] T 90223cd1477f46788581d2dcc937e162 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:28.854641  1574 consensus_queue.cc:1035] T 90223cd1477f46788581d2dcc937e162 P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.964450  1574 consensus_queue.cc:1035] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:28.970242  1574 consensus_queue.cc:1035] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.219684  1722 consensus_queue.cc:1035] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:29.226267  1733 consensus_queue.cc:1035] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:29.262955  1742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:29.263398  1742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:29.271299  1742 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'

INSERT report
    rows total: 2000
    time total: 291.304 ms
  time per row: 0.145652 ms
I20240503 02:11:29.708833   620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44878:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:29.717479  1399 tablet_service.cc:1467] Processing CreateTablet for tablet f7335d586927431f8ccdbbea16a8decb (DEFAULT_TABLE table=rebalance_test_table_3 [id=8f58f5e6adb445a7a8115f81c8666253]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:29.717664  1138 tablet_service.cc:1467] Processing CreateTablet for tablet f7335d586927431f8ccdbbea16a8decb (DEFAULT_TABLE table=rebalance_test_table_3 [id=8f58f5e6adb445a7a8115f81c8666253]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:29.717995  1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7335d586927431f8ccdbbea16a8decb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.718051  1138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7335d586927431f8ccdbbea16a8decb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.718531  1137 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca4c55ff01a428daaa3866840e9600a (DEFAULT_TABLE table=rebalance_test_table_3 [id=8f58f5e6adb445a7a8115f81c8666253]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:29.718940  1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ca4c55ff01a428daaa3866840e9600a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.719489   876 tablet_service.cc:1467] Processing CreateTablet for tablet f7335d586927431f8ccdbbea16a8decb (DEFAULT_TABLE table=rebalance_test_table_3 [id=8f58f5e6adb445a7a8115f81c8666253]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:29.719491   744 tablet_service.cc:1467] Processing CreateTablet for tablet 16edf1727a1e4c6f810d22bb97872613 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8f58f5e6adb445a7a8115f81c8666253]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:29.719902   876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7335d586927431f8ccdbbea16a8decb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.719944   744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16edf1727a1e4c6f810d22bb97872613. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.723125   875 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca4c55ff01a428daaa3866840e9600a (DEFAULT_TABLE table=rebalance_test_table_3 [id=8f58f5e6adb445a7a8115f81c8666253]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:29.723549   875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ca4c55ff01a428daaa3866840e9600a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.724785  1269 tablet_service.cc:1467] Processing CreateTablet for tablet 16edf1727a1e4c6f810d22bb97872613 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8f58f5e6adb445a7a8115f81c8666253]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:29.725203  1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16edf1727a1e4c6f810d22bb97872613. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.726872  1007 tablet_service.cc:1467] Processing CreateTablet for tablet 16edf1727a1e4c6f810d22bb97872613 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8f58f5e6adb445a7a8115f81c8666253]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:29.727315  1007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16edf1727a1e4c6f810d22bb97872613. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.727627  1400 tablet_service.cc:1467] Processing CreateTablet for tablet 3ca4c55ff01a428daaa3866840e9600a (DEFAULT_TABLE table=rebalance_test_table_3 [id=8f58f5e6adb445a7a8115f81c8666253]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:29.728031  1400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ca4c55ff01a428daaa3866840e9600a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:29.736946  1806 tablet_bootstrap.cc:492] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:29.741392  1806 tablet_bootstrap.cc:654] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.741499  1807 tablet_bootstrap.cc:492] T f7335d586927431f8ccdbbea16a8decb P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:29.743508  1807 tablet_bootstrap.cc:654] T f7335d586927431f8ccdbbea16a8decb P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.744153  1810 tablet_bootstrap.cc:492] T f7335d586927431f8ccdbbea16a8decb P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:29.747043  1810 tablet_bootstrap.cc:654] T f7335d586927431f8ccdbbea16a8decb P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.750213  1809 tablet_bootstrap.cc:492] T 16edf1727a1e4c6f810d22bb97872613 P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:29.749440  1808 tablet_bootstrap.cc:492] T 16edf1727a1e4c6f810d22bb97872613 P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:29.758771  1810 tablet_bootstrap.cc:492] T f7335d586927431f8ccdbbea16a8decb P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:29.759024  1810 ts_tablet_manager.cc:1367] T f7335d586927431f8ccdbbea16a8decb P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.015s	user 0.004s	sys 0.004s
I20240503 02:11:29.759133  1805 tablet_bootstrap.cc:492] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:29.759799  1810 raft_consensus.cc:348] T f7335d586927431f8ccdbbea16a8decb P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:29.759968  1810 raft_consensus.cc:374] T f7335d586927431f8ccdbbea16a8decb P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.760751  1810 raft_consensus.cc:729] T f7335d586927431f8ccdbbea16a8decb P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.761466  1806 tablet_bootstrap.cc:492] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:29.761584  1805 tablet_bootstrap.cc:654] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.761623  1806 ts_tablet_manager.cc:1367] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.025s	user 0.007s	sys 0.000s
I20240503 02:11:29.762401  1806 raft_consensus.cc:348] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:29.762565  1806 raft_consensus.cc:374] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.762713  1806 raft_consensus.cc:729] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.762991  1806 consensus_queue.cc:260] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:29.763119  1809 tablet_bootstrap.cc:654] T 16edf1727a1e4c6f810d22bb97872613 P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.763530  1806 ts_tablet_manager.cc:1398] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:29.763805  1806 tablet_bootstrap.cc:492] T 3ca4c55ff01a428daaa3866840e9600a P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:29.754550  1808 tablet_bootstrap.cc:654] T 16edf1727a1e4c6f810d22bb97872613 P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.765403  1810 consensus_queue.cc:260] T f7335d586927431f8ccdbbea16a8decb P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:29.766839  1810 ts_tablet_manager.cc:1398] T f7335d586927431f8ccdbbea16a8decb P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.000s
I20240503 02:11:29.767251  1806 tablet_bootstrap.cc:654] T 3ca4c55ff01a428daaa3866840e9600a P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.768469  1810 tablet_bootstrap.cc:492] T 3ca4c55ff01a428daaa3866840e9600a P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:29.770357  1810 tablet_bootstrap.cc:654] T 3ca4c55ff01a428daaa3866840e9600a P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.772017  1807 tablet_bootstrap.cc:492] T f7335d586927431f8ccdbbea16a8decb P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:29.772254  1807 ts_tablet_manager.cc:1367] T f7335d586927431f8ccdbbea16a8decb P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.031s	user 0.014s	sys 0.014s
I20240503 02:11:29.772276  1808 tablet_bootstrap.cc:492] T 16edf1727a1e4c6f810d22bb97872613 P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:29.772428  1808 ts_tablet_manager.cc:1367] T 16edf1727a1e4c6f810d22bb97872613 P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.023s	user 0.005s	sys 0.000s
I20240503 02:11:29.773106  1807 raft_consensus.cc:348] T f7335d586927431f8ccdbbea16a8decb P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:29.773212  1808 raft_consensus.cc:348] T 16edf1727a1e4c6f810d22bb97872613 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:29.773350  1807 raft_consensus.cc:374] T f7335d586927431f8ccdbbea16a8decb P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.773434  1807 raft_consensus.cc:729] T f7335d586927431f8ccdbbea16a8decb P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.773663  1807 consensus_queue.cc:260] T f7335d586927431f8ccdbbea16a8decb P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:29.774366  1809 tablet_bootstrap.cc:492] T 16edf1727a1e4c6f810d22bb97872613 P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:29.774629  1809 ts_tablet_manager.cc:1367] T 16edf1727a1e4c6f810d22bb97872613 P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.025s	user 0.005s	sys 0.000s
I20240503 02:11:29.775347  1807 ts_tablet_manager.cc:1398] T f7335d586927431f8ccdbbea16a8decb P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 02:11:29.775686  1807 tablet_bootstrap.cc:492] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:29.776348  1805 tablet_bootstrap.cc:492] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:29.775628  1809 raft_consensus.cc:348] T 16edf1727a1e4c6f810d22bb97872613 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:29.776899  1809 raft_consensus.cc:374] T 16edf1727a1e4c6f810d22bb97872613 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.777000  1809 raft_consensus.cc:729] T 16edf1727a1e4c6f810d22bb97872613 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.777248  1809 consensus_queue.cc:260] T 16edf1727a1e4c6f810d22bb97872613 P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:29.773425  1808 raft_consensus.cc:374] T 16edf1727a1e4c6f810d22bb97872613 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.777581  1808 raft_consensus.cc:729] T 16edf1727a1e4c6f810d22bb97872613 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.777848  1805 ts_tablet_manager.cc:1367] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.019s	user 0.002s	sys 0.003s
I20240503 02:11:29.778347  1807 tablet_bootstrap.cc:654] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:29.778503  1810 tablet_bootstrap.cc:492] T 3ca4c55ff01a428daaa3866840e9600a P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:29.778702  1810 ts_tablet_manager.cc:1367] T 3ca4c55ff01a428daaa3866840e9600a P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.003s
I20240503 02:11:29.778880  1805 raft_consensus.cc:348] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:29.779407  1808 consensus_queue.cc:260] T 16edf1727a1e4c6f810d22bb97872613 P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:29.779923  1810 raft_consensus.cc:348] T 3ca4c55ff01a428daaa3866840e9600a P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:29.780138  1810 raft_consensus.cc:374] T 3ca4c55ff01a428daaa3866840e9600a P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.780133  1809 ts_tablet_manager.cc:1398] T 16edf1727a1e4c6f810d22bb97872613 P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:11:29.780220  1810 raft_consensus.cc:729] T 3ca4c55ff01a428daaa3866840e9600a P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.780500  1810 consensus_queue.cc:260] T 3ca4c55ff01a428daaa3866840e9600a P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:29.780735  1808 ts_tablet_manager.cc:1398] T 16edf1727a1e4c6f810d22bb97872613 P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.000s
I20240503 02:11:29.781001  1805 raft_consensus.cc:374] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.781168  1805 raft_consensus.cc:729] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.781461  1805 consensus_queue.cc:260] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:29.781602  1806 tablet_bootstrap.cc:492] T 3ca4c55ff01a428daaa3866840e9600a P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:29.781776  1806 ts_tablet_manager.cc:1367] T 3ca4c55ff01a428daaa3866840e9600a P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.018s	user 0.002s	sys 0.004s
I20240503 02:11:29.782589  1806 raft_consensus.cc:348] T 3ca4c55ff01a428daaa3866840e9600a P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:29.782740  1805 ts_tablet_manager.cc:1398] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:11:29.782856  1806 raft_consensus.cc:374] T 3ca4c55ff01a428daaa3866840e9600a P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.782935  1806 raft_consensus.cc:729] T 3ca4c55ff01a428daaa3866840e9600a P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.783182  1806 consensus_queue.cc:260] T 3ca4c55ff01a428daaa3866840e9600a P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:29.783787  1806 ts_tablet_manager.cc:1398] T 3ca4c55ff01a428daaa3866840e9600a P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:29.784016  1807 tablet_bootstrap.cc:492] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:29.784179  1807 ts_tablet_manager.cc:1367] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.009s	user 0.001s	sys 0.001s
I20240503 02:11:29.784984  1810 ts_tablet_manager.cc:1398] T 3ca4c55ff01a428daaa3866840e9600a P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240503 02:11:29.785346  1807 raft_consensus.cc:348] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:29.785604  1807 raft_consensus.cc:374] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:29.785722  1807 raft_consensus.cc:729] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:29.786152  1807 consensus_queue.cc:260] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:29.786828  1807 ts_tablet_manager.cc:1398] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.001s
I20240503 02:11:29.803725  1611 raft_consensus.cc:482] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.803928  1611 raft_consensus.cc:504] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:29.804611  1611 leader_election.cc:288] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:29.805171  1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7335d586927431f8ccdbbea16a8decb" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e" is_pre_election: true
I20240503 02:11:29.805230  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7335d586927431f8ccdbbea16a8decb" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:29.805397  1420 raft_consensus.cc:2454] T f7335d586927431f8ccdbbea16a8decb P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 0.
I20240503 02:11:29.805488  1158 raft_consensus.cc:2454] T f7335d586927431f8ccdbbea16a8decb P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 0.
I20240503 02:11:29.805886   831 leader_election.cc:302] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:29.806131  1611 raft_consensus.cc:2781] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.806226  1611 raft_consensus.cc:482] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.806316  1611 raft_consensus.cc:3037] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.808163  1611 raft_consensus.cc:504] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:29.808714  1611 leader_election.cc:288] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 election: Requested vote from peers 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:29.809193  1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7335d586927431f8ccdbbea16a8decb" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e"
I20240503 02:11:29.809243  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7335d586927431f8ccdbbea16a8decb" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:29.809408  1420 raft_consensus.cc:3037] T f7335d586927431f8ccdbbea16a8decb P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.809453  1158 raft_consensus.cc:3037] T f7335d586927431f8ccdbbea16a8decb P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.811479  1158 raft_consensus.cc:2454] T f7335d586927431f8ccdbbea16a8decb P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 1.
I20240503 02:11:29.811486  1420 raft_consensus.cc:2454] T f7335d586927431f8ccdbbea16a8decb P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 1.
I20240503 02:11:29.812040   831 leader_election.cc:302] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:29.812369  1611 raft_consensus.cc:2781] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.812829  1611 raft_consensus.cc:686] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 LEADER]: Becoming Leader. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Running, Role: LEADER
I20240503 02:11:29.813127  1611 consensus_queue.cc:237] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:29.815976   618 catalog_manager.cc:5484] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2). New cstate: current_term: 1 leader_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:29.819450  1575 raft_consensus.cc:482] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.819625  1575 raft_consensus.cc:504] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:29.820322  1575 leader_election.cc:288] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511), 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189)
I20240503 02:11:29.820824  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16edf1727a1e4c6f810d22bb97872613" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1" is_pre_election: true
I20240503 02:11:29.820838  1026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16edf1727a1e4c6f810d22bb97872613" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f" is_pre_election: true
I20240503 02:11:29.821066  1026 raft_consensus.cc:2454] T 16edf1727a1e4c6f810d22bb97872613 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 0.
I20240503 02:11:29.821103  1289 raft_consensus.cc:2454] T 16edf1727a1e4c6f810d22bb97872613 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 0.
I20240503 02:11:29.821508   699 leader_election.cc:302] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, e8d9a43983f2478ea7ababa09b230177; no voters: 
I20240503 02:11:29.821805  1575 raft_consensus.cc:2781] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.821923  1575 raft_consensus.cc:482] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.822008  1575 raft_consensus.cc:3037] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.823897  1575 raft_consensus.cc:504] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:29.824421  1575 leader_election.cc:288] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 election: Requested vote from peers f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511), 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189)
I20240503 02:11:29.824913  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16edf1727a1e4c6f810d22bb97872613" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1"
I20240503 02:11:29.824882  1026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16edf1727a1e4c6f810d22bb97872613" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f"
I20240503 02:11:29.825124  1026 raft_consensus.cc:3037] T 16edf1727a1e4c6f810d22bb97872613 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.825214  1289 raft_consensus.cc:3037] T 16edf1727a1e4c6f810d22bb97872613 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.828014  1289 raft_consensus.cc:2454] T 16edf1727a1e4c6f810d22bb97872613 P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 1.
I20240503 02:11:29.828027  1026 raft_consensus.cc:2454] T 16edf1727a1e4c6f810d22bb97872613 P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 1.
I20240503 02:11:29.828466   699 leader_election.cc:302] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, e8d9a43983f2478ea7ababa09b230177; no voters: 
I20240503 02:11:29.828761  1575 raft_consensus.cc:2781] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.829144  1575 raft_consensus.cc:686] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [term 1 LEADER]: Becoming Leader. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Running, Role: LEADER
I20240503 02:11:29.829427  1575 consensus_queue.cc:237] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:29.832218   618 catalog_manager.cc:5484] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8d9a43983f2478ea7ababa09b230177 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "e8d9a43983f2478ea7ababa09b230177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:29.930266  1614 raft_consensus.cc:482] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:29.930470  1614 raft_consensus.cc:504] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:29.931216  1614 leader_election.cc:288] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135)
I20240503 02:11:29.931823  1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca4c55ff01a428daaa3866840e9600a" candidate_uuid: "288540692e7b404991570d9ca16ab6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e" is_pre_election: true
I20240503 02:11:29.932081  1420 raft_consensus.cc:2454] T 3ca4c55ff01a428daaa3866840e9600a P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 288540692e7b404991570d9ca16ab6c0 in term 0.
I20240503 02:11:29.932054   896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca4c55ff01a428daaa3866840e9600a" candidate_uuid: "288540692e7b404991570d9ca16ab6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" is_pre_election: true
I20240503 02:11:29.932315   896 raft_consensus.cc:2454] T 3ca4c55ff01a428daaa3866840e9600a P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 288540692e7b404991570d9ca16ab6c0 in term 0.
I20240503 02:11:29.932565  1093 leader_election.cc:302] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 4fda3b525c234f8f88aaefd355e7274e; no voters: 
I20240503 02:11:29.932847  1614 raft_consensus.cc:2781] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:29.932951  1614 raft_consensus.cc:482] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:29.933036  1614 raft_consensus.cc:3037] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.934927  1614 raft_consensus.cc:504] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:29.935449  1614 leader_election.cc:288] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [CANDIDATE]: Term 1 election: Requested vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135)
I20240503 02:11:29.935992   896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca4c55ff01a428daaa3866840e9600a" candidate_uuid: "288540692e7b404991570d9ca16ab6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
I20240503 02:11:29.936093  1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ca4c55ff01a428daaa3866840e9600a" candidate_uuid: "288540692e7b404991570d9ca16ab6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e"
I20240503 02:11:29.936183   896 raft_consensus.cc:3037] T 3ca4c55ff01a428daaa3866840e9600a P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.936285  1420 raft_consensus.cc:3037] T 3ca4c55ff01a428daaa3866840e9600a P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:29.938052  1420 raft_consensus.cc:2454] T 3ca4c55ff01a428daaa3866840e9600a P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 288540692e7b404991570d9ca16ab6c0 in term 1.
I20240503 02:11:29.938066   896 raft_consensus.cc:2454] T 3ca4c55ff01a428daaa3866840e9600a P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 288540692e7b404991570d9ca16ab6c0 in term 1.
I20240503 02:11:29.938467  1093 leader_election.cc:302] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:29.938804  1614 raft_consensus.cc:2781] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:29.939167  1614 raft_consensus.cc:686] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [term 1 LEADER]: Becoming Leader. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Running, Role: LEADER
I20240503 02:11:29.939448  1614 consensus_queue.cc:237] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:29.942102   618 catalog_manager.cc:5484] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 288540692e7b404991570d9ca16ab6c0 (127.0.145.4). New cstate: current_term: 1 leader_uuid: "288540692e7b404991570d9ca16ab6c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:30.295142  1822 consensus_queue.cc:1035] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.301242  1822 consensus_queue.cc:1035] T 16edf1727a1e4c6f810d22bb97872613 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:30.325604  1826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:30.326036  1826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:30.333864  1826 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 02:11:30.382006  1611 consensus_queue.cc:1035] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.393478  1611 consensus_queue.cc:1035] T f7335d586927431f8ccdbbea16a8decb P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.408510  1614 consensus_queue.cc:1035] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:30.416005  1824 consensus_queue.cc:1035] T 3ca4c55ff01a428daaa3866840e9600a P 288540692e7b404991570d9ca16ab6c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 289.269 ms
  time per row: 0.144635 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240503 02:11:31.716884   622 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44878:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:31.726354   744 tablet_service.cc:1467] Processing CreateTablet for tablet 3e64997bb292442eb4982967594bd7f7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8533f8dc97cf413a825640a7056fec51]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:31.726465  1007 tablet_service.cc:1467] Processing CreateTablet for tablet 3e64997bb292442eb4982967594bd7f7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8533f8dc97cf413a825640a7056fec51]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:31.726799   744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e64997bb292442eb4982967594bd7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:31.727073  1007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e64997bb292442eb4982967594bd7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:31.727855   743 tablet_service.cc:1467] Processing CreateTablet for tablet e9eaf1bda2e047f1bdbf7783d0ebb989 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8533f8dc97cf413a825640a7056fec51]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:31.728233   743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9eaf1bda2e047f1bdbf7783d0ebb989. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:31.728566  1269 tablet_service.cc:1467] Processing CreateTablet for tablet 3e64997bb292442eb4982967594bd7f7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8533f8dc97cf413a825640a7056fec51]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:31.728967  1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e64997bb292442eb4982967594bd7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:31.731804  1138 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6cdecb3c1d45dc8775f47bdd265d3c (DEFAULT_TABLE table=rebalance_test_table_4 [id=8533f8dc97cf413a825640a7056fec51]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:31.732239  1138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6cdecb3c1d45dc8775f47bdd265d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:31.734975  1268 tablet_service.cc:1467] Processing CreateTablet for tablet e9eaf1bda2e047f1bdbf7783d0ebb989 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8533f8dc97cf413a825640a7056fec51]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:31.735596   875 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6cdecb3c1d45dc8775f47bdd265d3c (DEFAULT_TABLE table=rebalance_test_table_4 [id=8533f8dc97cf413a825640a7056fec51]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:31.736006   875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6cdecb3c1d45dc8775f47bdd265d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:31.739282  1399 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6cdecb3c1d45dc8775f47bdd265d3c (DEFAULT_TABLE table=rebalance_test_table_4 [id=8533f8dc97cf413a825640a7056fec51]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:31.739346  1006 tablet_service.cc:1467] Processing CreateTablet for tablet e9eaf1bda2e047f1bdbf7783d0ebb989 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8533f8dc97cf413a825640a7056fec51]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:31.739792  1006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9eaf1bda2e047f1bdbf7783d0ebb989. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:31.739837  1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6cdecb3c1d45dc8775f47bdd265d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:31.747583  1268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9eaf1bda2e047f1bdbf7783d0ebb989. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:31.752691  1899 tablet_bootstrap.cc:492] T 3e64997bb292442eb4982967594bd7f7 P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:31.757329  1901 tablet_bootstrap.cc:492] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:31.757885  1897 tablet_bootstrap.cc:492] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:31.758651  1898 tablet_bootstrap.cc:492] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:31.758911  1899 tablet_bootstrap.cc:654] T 3e64997bb292442eb4982967594bd7f7 P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:31.759208  1900 tablet_bootstrap.cc:492] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:31.759871  1901 tablet_bootstrap.cc:654] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:31.762199  1897 tablet_bootstrap.cc:654] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:31.762920  1898 tablet_bootstrap.cc:654] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:31.763528  1900 tablet_bootstrap.cc:654] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:31.766239  1896 tablet_bootstrap.cc:492] T 3e64997bb292442eb4982967594bd7f7 P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:31.769551  1896 tablet_bootstrap.cc:654] T 3e64997bb292442eb4982967594bd7f7 P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:31.772289  1901 tablet_bootstrap.cc:492] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:31.772487  1901 ts_tablet_manager.cc:1367] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.002s
I20240503 02:11:31.773515  1901 raft_consensus.cc:348] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:31.773792  1901 raft_consensus.cc:374] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:31.773891  1901 raft_consensus.cc:729] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:31.774178  1901 consensus_queue.cc:260] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:31.777354  1901 ts_tablet_manager.cc:1398] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.001s
I20240503 02:11:31.777696  1897 tablet_bootstrap.cc:492] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:31.777738  1901 tablet_bootstrap.cc:492] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:31.777870  1897 ts_tablet_manager.cc:1367] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.020s	user 0.006s	sys 0.000s
I20240503 02:11:31.778735  1897 raft_consensus.cc:348] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:31.778967  1897 raft_consensus.cc:374] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:31.779021  1900 tablet_bootstrap.cc:492] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:31.779054  1897 raft_consensus.cc:729] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:31.778016  1896 tablet_bootstrap.cc:492] T 3e64997bb292442eb4982967594bd7f7 P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:31.779361  1897 consensus_queue.cc:260] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:31.779701  1901 tablet_bootstrap.cc:654] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:31.779734  1900 ts_tablet_manager.cc:1367] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.021s	user 0.003s	sys 0.004s
I20240503 02:11:31.779879  1896 ts_tablet_manager.cc:1367] T 3e64997bb292442eb4982967594bd7f7 P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.000s
I20240503 02:11:31.780122  1897 ts_tablet_manager.cc:1398] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:11:31.780974  1899 tablet_bootstrap.cc:492] T 3e64997bb292442eb4982967594bd7f7 P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:31.780920  1900 raft_consensus.cc:348] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:31.781235  1899 ts_tablet_manager.cc:1367] T 3e64997bb292442eb4982967594bd7f7 P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.029s	user 0.010s	sys 0.010s
I20240503 02:11:31.781813  1896 raft_consensus.cc:348] T 3e64997bb292442eb4982967594bd7f7 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:31.782086  1900 raft_consensus.cc:374] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:31.782222  1896 raft_consensus.cc:374] T 3e64997bb292442eb4982967594bd7f7 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:31.782420  1900 raft_consensus.cc:729] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:31.782378  1899 raft_consensus.cc:348] T 3e64997bb292442eb4982967594bd7f7 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:31.782706  1899 raft_consensus.cc:374] T 3e64997bb292442eb4982967594bd7f7 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:31.782577  1896 raft_consensus.cc:729] T 3e64997bb292442eb4982967594bd7f7 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:31.782907  1899 raft_consensus.cc:729] T 3e64997bb292442eb4982967594bd7f7 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:31.783283  1899 consensus_queue.cc:260] T 3e64997bb292442eb4982967594bd7f7 P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:31.783403  1896 consensus_queue.cc:260] T 3e64997bb292442eb4982967594bd7f7 P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:31.783031  1900 consensus_queue.cc:260] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:31.786068  1896 ts_tablet_manager.cc:1398] T 3e64997bb292442eb4982967594bd7f7 P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240503 02:11:31.786401  1896 tablet_bootstrap.cc:492] T e9eaf1bda2e047f1bdbf7783d0ebb989 P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:31.786743  1899 ts_tablet_manager.cc:1398] T 3e64997bb292442eb4982967594bd7f7 P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240503 02:11:31.787026  1900 ts_tablet_manager.cc:1398] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.001s
I20240503 02:11:31.787945  1899 tablet_bootstrap.cc:492] T e9eaf1bda2e047f1bdbf7783d0ebb989 P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:31.788703  1896 tablet_bootstrap.cc:654] T e9eaf1bda2e047f1bdbf7783d0ebb989 P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:31.790212  1899 tablet_bootstrap.cc:654] T e9eaf1bda2e047f1bdbf7783d0ebb989 P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:31.791394  1901 tablet_bootstrap.cc:492] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:31.791669  1901 ts_tablet_manager.cc:1367] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.014s	user 0.010s	sys 0.001s
I20240503 02:11:31.792789  1901 raft_consensus.cc:348] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:31.793192  1901 raft_consensus.cc:374] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:31.793385  1901 raft_consensus.cc:729] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:31.793787  1901 consensus_queue.cc:260] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:31.795467  1901 ts_tablet_manager.cc:1398] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:11:31.797104  1899 tablet_bootstrap.cc:492] T e9eaf1bda2e047f1bdbf7783d0ebb989 P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:31.797264  1899 ts_tablet_manager.cc:1367] T e9eaf1bda2e047f1bdbf7783d0ebb989 P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.008s
I20240503 02:11:31.797940  1899 raft_consensus.cc:348] T e9eaf1bda2e047f1bdbf7783d0ebb989 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:31.798231  1899 raft_consensus.cc:374] T e9eaf1bda2e047f1bdbf7783d0ebb989 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:31.798336  1899 raft_consensus.cc:729] T e9eaf1bda2e047f1bdbf7783d0ebb989 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:31.798563  1899 consensus_queue.cc:260] T e9eaf1bda2e047f1bdbf7783d0ebb989 P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:31.799885  1899 ts_tablet_manager.cc:1398] T e9eaf1bda2e047f1bdbf7783d0ebb989 P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:31.800400  1898 tablet_bootstrap.cc:492] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:31.800558  1898 ts_tablet_manager.cc:1367] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.042s	user 0.006s	sys 0.000s
I20240503 02:11:31.801331  1898 raft_consensus.cc:348] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:31.801493  1898 raft_consensus.cc:374] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:31.801584  1898 raft_consensus.cc:729] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:31.801791  1898 consensus_queue.cc:260] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:31.802305  1898 ts_tablet_manager.cc:1398] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:11:31.803144  1896 tablet_bootstrap.cc:492] T e9eaf1bda2e047f1bdbf7783d0ebb989 P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:31.803320  1896 ts_tablet_manager.cc:1367] T e9eaf1bda2e047f1bdbf7783d0ebb989 P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.017s	user 0.009s	sys 0.004s
I20240503 02:11:31.803999  1896 raft_consensus.cc:348] T e9eaf1bda2e047f1bdbf7783d0ebb989 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:31.804181  1896 raft_consensus.cc:374] T e9eaf1bda2e047f1bdbf7783d0ebb989 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:31.804263  1896 raft_consensus.cc:729] T e9eaf1bda2e047f1bdbf7783d0ebb989 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:31.804514  1896 consensus_queue.cc:260] T e9eaf1bda2e047f1bdbf7783d0ebb989 P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:31.805008  1896 ts_tablet_manager.cc:1398] T e9eaf1bda2e047f1bdbf7783d0ebb989 P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:11:31.825963  1677 raft_consensus.cc:482] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:31.826156  1677 raft_consensus.cc:504] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:31.826876  1677 leader_election.cc:288] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189), e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401)
I20240503 02:11:31.827284  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9eaf1bda2e047f1bdbf7783d0ebb989" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1" is_pre_election: true
I20240503 02:11:31.827419   764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9eaf1bda2e047f1bdbf7783d0ebb989" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177" is_pre_election: true
I20240503 02:11:31.827512  1288 raft_consensus.cc:2454] T e9eaf1bda2e047f1bdbf7783d0ebb989 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 0.
I20240503 02:11:31.827647   764 raft_consensus.cc:2454] T e9eaf1bda2e047f1bdbf7783d0ebb989 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 0.
I20240503 02:11:31.827945   962 leader_election.cc:302] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:31.828200  1677 raft_consensus.cc:2781] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:31.828284  1677 raft_consensus.cc:482] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:31.828384  1677 raft_consensus.cc:3037] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:31.830230  1677 raft_consensus.cc:504] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:31.830857  1677 leader_election.cc:288] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 election: Requested vote from peers 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189), e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401)
I20240503 02:11:31.831269  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9eaf1bda2e047f1bdbf7783d0ebb989" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1"
I20240503 02:11:31.831326   764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9eaf1bda2e047f1bdbf7783d0ebb989" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177"
I20240503 02:11:31.831454  1288 raft_consensus.cc:3037] T e9eaf1bda2e047f1bdbf7783d0ebb989 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:31.831490   764 raft_consensus.cc:3037] T e9eaf1bda2e047f1bdbf7783d0ebb989 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:31.833320   764 raft_consensus.cc:2454] T e9eaf1bda2e047f1bdbf7783d0ebb989 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 1.
I20240503 02:11:31.833328  1288 raft_consensus.cc:2454] T e9eaf1bda2e047f1bdbf7783d0ebb989 P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 1.
I20240503 02:11:31.833788   961 leader_election.cc:302] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8d9a43983f2478ea7ababa09b230177, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:31.834087  1677 raft_consensus.cc:2781] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:31.834483  1677 raft_consensus.cc:686] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [term 1 LEADER]: Becoming Leader. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Running, Role: LEADER
I20240503 02:11:31.835106  1677 consensus_queue.cc:237] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } }
I20240503 02:11:31.838004   617 catalog_manager.cc:5484] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f reported cstate change: term changed from 0 to 1, leader changed from <none> to f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3). New cstate: current_term: 1 leader_uuid: "f5e38dcb24084a1e969f83bf34aef88f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:31.841467  1677 raft_consensus.cc:482] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:31.841619  1677 raft_consensus.cc:504] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:31.842188  1677 leader_election.cc:288] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401), 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189)
I20240503 02:11:31.842547   764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e64997bb292442eb4982967594bd7f7" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177" is_pre_election: true
I20240503 02:11:31.842687  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e64997bb292442eb4982967594bd7f7" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1" is_pre_election: true
I20240503 02:11:31.842810   764 raft_consensus.cc:2454] T 3e64997bb292442eb4982967594bd7f7 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 0.
I20240503 02:11:31.842917  1288 raft_consensus.cc:2454] T 3e64997bb292442eb4982967594bd7f7 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 0.
I20240503 02:11:31.843174   961 leader_election.cc:302] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8d9a43983f2478ea7ababa09b230177, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:31.843456  1677 raft_consensus.cc:2781] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:31.843573  1677 raft_consensus.cc:482] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:31.843657  1677 raft_consensus.cc:3037] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:31.845670  1677 raft_consensus.cc:504] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:31.846127  1677 leader_election.cc:288] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 election: Requested vote from peers e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401), 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189)
I20240503 02:11:31.846547   764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e64997bb292442eb4982967594bd7f7" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177"
I20240503 02:11:31.846623  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e64997bb292442eb4982967594bd7f7" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1"
I20240503 02:11:31.846763   764 raft_consensus.cc:3037] T 3e64997bb292442eb4982967594bd7f7 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:31.846858  1288 raft_consensus.cc:3037] T 3e64997bb292442eb4982967594bd7f7 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:31.848858   764 raft_consensus.cc:2454] T 3e64997bb292442eb4982967594bd7f7 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 1.
I20240503 02:11:31.848910  1288 raft_consensus.cc:2454] T 3e64997bb292442eb4982967594bd7f7 P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 1.
I20240503 02:11:31.849396   962 leader_election.cc:302] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:31.849712  1677 raft_consensus.cc:2781] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:31.849886  1677 raft_consensus.cc:686] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [term 1 LEADER]: Becoming Leader. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Running, Role: LEADER
I20240503 02:11:31.850179  1677 consensus_queue.cc:237] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:31.852844   617 catalog_manager.cc:5484] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f reported cstate change: term changed from 0 to 1, leader changed from <none> to f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3). New cstate: current_term: 1 leader_uuid: "f5e38dcb24084a1e969f83bf34aef88f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:32.146679  1722 raft_consensus.cc:482] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:32.146895  1722 raft_consensus.cc:504] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:32.147562  1722 leader_election.cc:288] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:32.148198   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6cdecb3c1d45dc8775f47bdd265d3c" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" is_pre_election: true
I20240503 02:11:32.148221  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6cdecb3c1d45dc8775f47bdd265d3c" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:32.148415   895 raft_consensus.cc:2454] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:32.148495  1158 raft_consensus.cc:2454] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:32.148905  1355 leader_election.cc:302] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:32.149206  1722 raft_consensus.cc:2781] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:32.149334  1722 raft_consensus.cc:482] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:32.149446  1722 raft_consensus.cc:3037] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:32.151412  1722 raft_consensus.cc:504] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:32.151930  1722 leader_election.cc:288] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Requested vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:32.152418   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6cdecb3c1d45dc8775f47bdd265d3c" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
I20240503 02:11:32.152422  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6cdecb3c1d45dc8775f47bdd265d3c" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:32.152601   895 raft_consensus.cc:3037] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:32.152603  1158 raft_consensus.cc:3037] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:32.154490  1158 raft_consensus.cc:2454] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:32.154510   895 raft_consensus.cc:2454] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:32.154887  1355 leader_election.cc:302] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:32.155158  1722 raft_consensus.cc:2781] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:32.155550  1722 raft_consensus.cc:686] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [term 1 LEADER]: Becoming Leader. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Running, Role: LEADER
I20240503 02:11:32.155830  1722 consensus_queue.cc:237] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:32.158578   617 catalog_manager.cc:5484] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6). New cstate: current_term: 1 leader_uuid: "4fda3b525c234f8f88aaefd355e7274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:32.404425  1677 consensus_queue.cc:1035] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:32.411417  1918 consensus_queue.cc:1035] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:32.412951  1919 consensus_queue.cc:1035] T 3e64997bb292442eb4982967594bd7f7 P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:32.420818  1677 consensus_queue.cc:1035] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:32.550336  1916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:32.550735  1916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:32.563099  1916 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 02:11:32.672912  1722 consensus_queue.cc:1035] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:32.702963  1950 consensus_queue.cc:1035] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 279.033 ms
  time per row: 0.139516 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      4       1192 libcrypto.so.1.1
-----------------------------------------------------

I20240503 02:11:33.943912   622 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44878:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:33.953950  1399 tablet_service.cc:1467] Processing CreateTablet for tablet 41a6042ee7784967913fbf65d25ddc28 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2e3ca6be174248d39cbf7747d059aa62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:33.954383  1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a6042ee7784967913fbf65d25ddc28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:33.954977   875 tablet_service.cc:1467] Processing CreateTablet for tablet 41a6042ee7784967913fbf65d25ddc28 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2e3ca6be174248d39cbf7747d059aa62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:33.955392   875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a6042ee7784967913fbf65d25ddc28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:33.955302  1138 tablet_service.cc:1467] Processing CreateTablet for tablet 41a6042ee7784967913fbf65d25ddc28 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2e3ca6be174248d39cbf7747d059aa62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:33.955679  1138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a6042ee7784967913fbf65d25ddc28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:33.958678  1137 tablet_service.cc:1467] Processing CreateTablet for tablet 9095a299c3084a30b4ec750b5883ff9c (DEFAULT_TABLE table=rebalance_test_table_5 [id=2e3ca6be174248d39cbf7747d059aa62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:33.959219  1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9095a299c3084a30b4ec750b5883ff9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:33.965955  1983 tablet_bootstrap.cc:492] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:33.967752  1983 tablet_bootstrap.cc:654] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:33.970511  1984 tablet_bootstrap.cc:492] T 41a6042ee7784967913fbf65d25ddc28 P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:33.972836  1984 tablet_bootstrap.cc:654] T 41a6042ee7784967913fbf65d25ddc28 P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:33.973366  1982 tablet_bootstrap.cc:492] T 41a6042ee7784967913fbf65d25ddc28 P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:33.974681  1983 tablet_bootstrap.cc:492] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:33.974869  1983 ts_tablet_manager.cc:1367] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.001s
I20240503 02:11:33.975090  1982 tablet_bootstrap.cc:654] T 41a6042ee7784967913fbf65d25ddc28 P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:33.975049   876 tablet_service.cc:1467] Processing CreateTablet for tablet 9095a299c3084a30b4ec750b5883ff9c (DEFAULT_TABLE table=rebalance_test_table_5 [id=2e3ca6be174248d39cbf7747d059aa62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:33.975454   876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9095a299c3084a30b4ec750b5883ff9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:33.975792  1983 raft_consensus.cc:348] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:33.976104  1983 raft_consensus.cc:374] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:33.976261  1983 raft_consensus.cc:729] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:33.976567  1983 consensus_queue.cc:260] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:33.977097  1982 tablet_bootstrap.cc:492] T 41a6042ee7784967913fbf65d25ddc28 P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:33.977211  1983 ts_tablet_manager.cc:1398] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:11:33.977275  1982 ts_tablet_manager.cc:1367] T 41a6042ee7784967913fbf65d25ddc28 P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.003s
I20240503 02:11:33.978085  1982 raft_consensus.cc:348] T 41a6042ee7784967913fbf65d25ddc28 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:33.978250  1982 raft_consensus.cc:374] T 41a6042ee7784967913fbf65d25ddc28 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:33.978358  1982 raft_consensus.cc:729] T 41a6042ee7784967913fbf65d25ddc28 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:33.978673  1982 consensus_queue.cc:260] T 41a6042ee7784967913fbf65d25ddc28 P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:33.979252  1982 ts_tablet_manager.cc:1398] T 41a6042ee7784967913fbf65d25ddc28 P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:11:33.980719  1399 tablet_service.cc:1467] Processing CreateTablet for tablet 9095a299c3084a30b4ec750b5883ff9c (DEFAULT_TABLE table=rebalance_test_table_5 [id=2e3ca6be174248d39cbf7747d059aa62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:33.981165  1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9095a299c3084a30b4ec750b5883ff9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:33.981627  1983 tablet_bootstrap.cc:492] T 9095a299c3084a30b4ec750b5883ff9c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:33.983371  1983 tablet_bootstrap.cc:654] T 9095a299c3084a30b4ec750b5883ff9c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:33.984226  1007 tablet_service.cc:1467] Processing CreateTablet for tablet b667f314948a4c2c917cc56dde38c67c (DEFAULT_TABLE table=rebalance_test_table_5 [id=2e3ca6be174248d39cbf7747d059aa62]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:33.984301  1268 tablet_service.cc:1467] Processing CreateTablet for tablet b667f314948a4c2c917cc56dde38c67c (DEFAULT_TABLE table=rebalance_test_table_5 [id=2e3ca6be174248d39cbf7747d059aa62]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:33.984666  1007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b667f314948a4c2c917cc56dde38c67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:33.984678  1268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b667f314948a4c2c917cc56dde38c67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:33.987295   744 tablet_service.cc:1467] Processing CreateTablet for tablet b667f314948a4c2c917cc56dde38c67c (DEFAULT_TABLE table=rebalance_test_table_5 [id=2e3ca6be174248d39cbf7747d059aa62]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:33.987576  1984 tablet_bootstrap.cc:492] T 41a6042ee7784967913fbf65d25ddc28 P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:33.987751   744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b667f314948a4c2c917cc56dde38c67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:33.987774  1984 ts_tablet_manager.cc:1367] T 41a6042ee7784967913fbf65d25ddc28 P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.017s	user 0.003s	sys 0.000s
I20240503 02:11:33.988902  1984 raft_consensus.cc:348] T 41a6042ee7784967913fbf65d25ddc28 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:33.989226  1984 raft_consensus.cc:374] T 41a6042ee7784967913fbf65d25ddc28 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:33.989362  1984 raft_consensus.cc:729] T 41a6042ee7784967913fbf65d25ddc28 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:33.989682  1984 consensus_queue.cc:260] T 41a6042ee7784967913fbf65d25ddc28 P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:33.990551  1984 ts_tablet_manager.cc:1398] T 41a6042ee7784967913fbf65d25ddc28 P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:11:33.990923  1984 tablet_bootstrap.cc:492] T 9095a299c3084a30b4ec750b5883ff9c P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:33.993088  1984 tablet_bootstrap.cc:654] T 9095a299c3084a30b4ec750b5883ff9c P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:33.995580  1982 tablet_bootstrap.cc:492] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:33.997308  1982 tablet_bootstrap.cc:654] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.019719  1993 tablet_bootstrap.cc:492] T b667f314948a4c2c917cc56dde38c67c P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:34.023959  1611 raft_consensus.cc:482] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:34.024192  1611 raft_consensus.cc:504] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:34.024919  1611 leader_election.cc:288] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:34.026506  1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a6042ee7784967913fbf65d25ddc28" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e" is_pre_election: true
I20240503 02:11:34.026798  1420 raft_consensus.cc:2454] T 41a6042ee7784967913fbf65d25ddc28 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 0.
I20240503 02:11:34.027343   831 leader_election.cc:302] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:34.027653  1992 tablet_bootstrap.cc:492] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:34.027637  1611 raft_consensus.cc:2781] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:34.027736  1611 raft_consensus.cc:482] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:34.027856  1611 raft_consensus.cc:3037] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.028054  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a6042ee7784967913fbf65d25ddc28" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:34.028306  1158 raft_consensus.cc:2454] T 41a6042ee7784967913fbf65d25ddc28 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 0.
I20240503 02:11:34.029436  1993 tablet_bootstrap.cc:654] T b667f314948a4c2c917cc56dde38c67c P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.030019  1992 tablet_bootstrap.cc:654] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.030690  1611 raft_consensus.cc:504] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:34.032297  1611 leader_election.cc:288] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 election: Requested vote from peers 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:34.032584  1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a6042ee7784967913fbf65d25ddc28" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e"
I20240503 02:11:34.032848  1420 raft_consensus.cc:3037] T 41a6042ee7784967913fbf65d25ddc28 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.034817  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a6042ee7784967913fbf65d25ddc28" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:34.035015  1158 raft_consensus.cc:3037] T 41a6042ee7784967913fbf65d25ddc28 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.035390  1990 tablet_bootstrap.cc:492] T b667f314948a4c2c917cc56dde38c67c P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:34.035698  1982 tablet_bootstrap.cc:492] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:34.035913  1982 ts_tablet_manager.cc:1367] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.040s	user 0.004s	sys 0.014s
I20240503 02:11:34.036748  1982 raft_consensus.cc:348] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:34.037011  1982 raft_consensus.cc:374] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.037138  1982 raft_consensus.cc:729] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.037425  1982 consensus_queue.cc:260] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:34.037626  1990 tablet_bootstrap.cc:654] T b667f314948a4c2c917cc56dde38c67c P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.037850  1158 raft_consensus.cc:2454] T 41a6042ee7784967913fbf65d25ddc28 P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 1.
I20240503 02:11:34.038064  1982 ts_tablet_manager.cc:1398] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 02:11:34.038471   831 leader_election.cc:302] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:34.039184  1611 raft_consensus.cc:2781] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:34.040804  1992 tablet_bootstrap.cc:492] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:34.041015  1992 ts_tablet_manager.cc:1367] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.013s	user 0.000s	sys 0.003s
I20240503 02:11:34.041605  1420 raft_consensus.cc:2454] T 41a6042ee7784967913fbf65d25ddc28 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 1.
I20240503 02:11:34.042212  1992 raft_consensus.cc:348] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:34.042541  1992 raft_consensus.cc:374] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.042959  1992 raft_consensus.cc:729] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.046829  1992 consensus_queue.cc:260] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:34.047560  1992 ts_tablet_manager.cc:1398] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.002s
I20240503 02:11:34.053875  1983 tablet_bootstrap.cc:492] T 9095a299c3084a30b4ec750b5883ff9c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:34.054112  1611 raft_consensus.cc:686] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 LEADER]: Becoming Leader. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Running, Role: LEADER
I20240503 02:11:34.054173  1983 ts_tablet_manager.cc:1367] T 9095a299c3084a30b4ec750b5883ff9c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.073s	user 0.008s	sys 0.024s
I20240503 02:11:34.054469  1611 consensus_queue.cc:237] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:34.055219  1983 raft_consensus.cc:348] T 9095a299c3084a30b4ec750b5883ff9c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:34.055451  1983 raft_consensus.cc:374] T 9095a299c3084a30b4ec750b5883ff9c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.055552  1983 raft_consensus.cc:729] T 9095a299c3084a30b4ec750b5883ff9c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.057168  1993 tablet_bootstrap.cc:492] T b667f314948a4c2c917cc56dde38c67c P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:34.057401  1993 ts_tablet_manager.cc:1367] T b667f314948a4c2c917cc56dde38c67c P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.038s	user 0.005s	sys 0.000s
I20240503 02:11:34.057919   622 catalog_manager.cc:5484] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2). New cstate: current_term: 1 leader_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:34.058254  1983 consensus_queue.cc:260] T 9095a299c3084a30b4ec750b5883ff9c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:34.058925  1983 ts_tablet_manager.cc:1398] T 9095a299c3084a30b4ec750b5883ff9c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:11:34.059370  1993 raft_consensus.cc:348] T b667f314948a4c2c917cc56dde38c67c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:34.059603  1993 raft_consensus.cc:374] T b667f314948a4c2c917cc56dde38c67c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.059728  1993 raft_consensus.cc:729] T b667f314948a4c2c917cc56dde38c67c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.060003  1993 consensus_queue.cc:260] T b667f314948a4c2c917cc56dde38c67c P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:34.060674  1993 ts_tablet_manager.cc:1398] T b667f314948a4c2c917cc56dde38c67c P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:34.060102  1990 tablet_bootstrap.cc:492] T b667f314948a4c2c917cc56dde38c67c P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:34.061386  1990 ts_tablet_manager.cc:1367] T b667f314948a4c2c917cc56dde38c67c P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.026s	user 0.004s	sys 0.000s
I20240503 02:11:34.062187  1990 raft_consensus.cc:348] T b667f314948a4c2c917cc56dde38c67c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:34.062434  1990 raft_consensus.cc:374] T b667f314948a4c2c917cc56dde38c67c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.062554  1990 raft_consensus.cc:729] T b667f314948a4c2c917cc56dde38c67c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.062824  1990 consensus_queue.cc:260] T b667f314948a4c2c917cc56dde38c67c P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:34.063575  1990 ts_tablet_manager.cc:1398] T b667f314948a4c2c917cc56dde38c67c P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:34.064335  1984 tablet_bootstrap.cc:492] T 9095a299c3084a30b4ec750b5883ff9c P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:34.064525  1984 ts_tablet_manager.cc:1367] T 9095a299c3084a30b4ec750b5883ff9c P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.074s	user 0.011s	sys 0.026s
I20240503 02:11:34.065449  1984 raft_consensus.cc:348] T 9095a299c3084a30b4ec750b5883ff9c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:34.065675  1984 raft_consensus.cc:374] T 9095a299c3084a30b4ec750b5883ff9c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.065758  1984 raft_consensus.cc:729] T 9095a299c3084a30b4ec750b5883ff9c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.066026  1984 consensus_queue.cc:260] T 9095a299c3084a30b4ec750b5883ff9c P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:34.066557  1984 ts_tablet_manager.cc:1398] T 9095a299c3084a30b4ec750b5883ff9c P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:11:34.076124  1950 raft_consensus.cc:482] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:34.076336  1950 raft_consensus.cc:504] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:34.076918  1950 leader_election.cc:288] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:34.077409   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9095a299c3084a30b4ec750b5883ff9c" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" is_pre_election: true
I20240503 02:11:34.077463  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9095a299c3084a30b4ec750b5883ff9c" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:34.077687   895 raft_consensus.cc:2454] T 9095a299c3084a30b4ec750b5883ff9c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:34.077708  1158 raft_consensus.cc:2454] T 9095a299c3084a30b4ec750b5883ff9c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:34.078138  1355 leader_election.cc:302] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:34.078421  1950 raft_consensus.cc:2781] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:34.078512  1950 raft_consensus.cc:482] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:34.078639  1950 raft_consensus.cc:3037] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.080749  1950 raft_consensus.cc:504] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:34.081333  1950 leader_election.cc:288] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Requested vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:34.081847   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9095a299c3084a30b4ec750b5883ff9c" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
I20240503 02:11:34.081864  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9095a299c3084a30b4ec750b5883ff9c" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:34.082078  1158 raft_consensus.cc:3037] T 9095a299c3084a30b4ec750b5883ff9c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.082078   895 raft_consensus.cc:3037] T 9095a299c3084a30b4ec750b5883ff9c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.084949  1158 raft_consensus.cc:2454] T 9095a299c3084a30b4ec750b5883ff9c P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:34.084955   895 raft_consensus.cc:2454] T 9095a299c3084a30b4ec750b5883ff9c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:34.085438  1355 leader_election.cc:302] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:34.085707  1950 raft_consensus.cc:2781] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:34.086107  1950 raft_consensus.cc:686] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [term 1 LEADER]: Becoming Leader. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Running, Role: LEADER
I20240503 02:11:34.086436  1950 consensus_queue.cc:237] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:34.089401   622 catalog_manager.cc:5484] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6). New cstate: current_term: 1 leader_uuid: "4fda3b525c234f8f88aaefd355e7274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:34.247100  1891 raft_consensus.cc:482] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:34.247288  1891 raft_consensus.cc:504] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:34.247947  1891 leader_election.cc:288] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511), e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401)
I20240503 02:11:34.248872   763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b667f314948a4c2c917cc56dde38c67c" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177" is_pre_election: true
I20240503 02:11:34.248881  1026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b667f314948a4c2c917cc56dde38c67c" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f" is_pre_election: true
I20240503 02:11:34.249128   763 raft_consensus.cc:2454] T b667f314948a4c2c917cc56dde38c67c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 0.
I20240503 02:11:34.249171  1026 raft_consensus.cc:2454] T b667f314948a4c2c917cc56dde38c67c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 0.
I20240503 02:11:34.249557  1223 leader_election.cc:302] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, e8d9a43983f2478ea7ababa09b230177; no voters: 
I20240503 02:11:34.249794  1891 raft_consensus.cc:2781] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:34.249904  1891 raft_consensus.cc:482] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:34.249996  1891 raft_consensus.cc:3037] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.252056  1891 raft_consensus.cc:504] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:34.252633  1891 leader_election.cc:288] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 election: Requested vote from peers f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511), e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401)
I20240503 02:11:34.253093  1026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b667f314948a4c2c917cc56dde38c67c" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f"
I20240503 02:11:34.253317  1026 raft_consensus.cc:3037] T b667f314948a4c2c917cc56dde38c67c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.253325   763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b667f314948a4c2c917cc56dde38c67c" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177"
I20240503 02:11:34.253540   763 raft_consensus.cc:3037] T b667f314948a4c2c917cc56dde38c67c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.255502  1026 raft_consensus.cc:2454] T b667f314948a4c2c917cc56dde38c67c P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 1.
I20240503 02:11:34.255684   763 raft_consensus.cc:2454] T b667f314948a4c2c917cc56dde38c67c P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 1.
I20240503 02:11:34.255903  1223 leader_election.cc:302] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:34.256201  1891 raft_consensus.cc:2781] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:34.256348  1891 raft_consensus.cc:686] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [term 1 LEADER]: Becoming Leader. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Running, Role: LEADER
I20240503 02:11:34.256608  1891 consensus_queue.cc:237] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:34.259254   622 catalog_manager.cc:5484] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5). New cstate: current_term: 1 leader_uuid: "572709bd9ae041a4a98029298a6f69c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:34.516296  1996 consensus_queue.cc:1035] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:34.530499  2007 consensus_queue.cc:1035] T 41a6042ee7784967913fbf65d25ddc28 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:34.553238  1950 consensus_queue.cc:1035] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:34.564590  1950 consensus_queue.cc:1035] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:34.658690  2003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:34.659008  2003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:34.667207  2003 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 02:11:34.751559  1891 consensus_queue.cc:1035] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:34.785663  1891 consensus_queue.cc:1035] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 261.832 ms
  time per row: 0.130916 ms
I20240503 02:11:35.106420   618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44878:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:35.115465   744 tablet_service.cc:1467] Processing CreateTablet for tablet 2686a2ecc7594849ba4f0213764f27a8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=729bd25784924c04aa5f6de9d13a45d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.115769  1268 tablet_service.cc:1467] Processing CreateTablet for tablet 2686a2ecc7594849ba4f0213764f27a8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=729bd25784924c04aa5f6de9d13a45d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.115964   744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2686a2ecc7594849ba4f0213764f27a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.116209  1268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2686a2ecc7594849ba4f0213764f27a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.116314  1137 tablet_service.cc:1467] Processing CreateTablet for tablet 7e7f9eb8403e45d88a5532506d06e2e2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=729bd25784924c04aa5f6de9d13a45d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.116721  1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e7f9eb8403e45d88a5532506d06e2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.118184  1399 tablet_service.cc:1467] Processing CreateTablet for tablet 7e7f9eb8403e45d88a5532506d06e2e2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=729bd25784924c04aa5f6de9d13a45d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.118584  1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e7f9eb8403e45d88a5532506d06e2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.119035   743 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9396ad859e4ea08a06ef08570ac3fb (DEFAULT_TABLE table=rebalance_test_table_6 [id=729bd25784924c04aa5f6de9d13a45d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.119415   743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9396ad859e4ea08a06ef08570ac3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.123306  1007 tablet_service.cc:1467] Processing CreateTablet for tablet 2686a2ecc7594849ba4f0213764f27a8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=729bd25784924c04aa5f6de9d13a45d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.123354  1269 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9396ad859e4ea08a06ef08570ac3fb (DEFAULT_TABLE table=rebalance_test_table_6 [id=729bd25784924c04aa5f6de9d13a45d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.123414  1006 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9396ad859e4ea08a06ef08570ac3fb (DEFAULT_TABLE table=rebalance_test_table_6 [id=729bd25784924c04aa5f6de9d13a45d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.123754  1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9396ad859e4ea08a06ef08570ac3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.123775  1007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2686a2ecc7594849ba4f0213764f27a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.123981  1006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9396ad859e4ea08a06ef08570ac3fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.127563   876 tablet_service.cc:1467] Processing CreateTablet for tablet 7e7f9eb8403e45d88a5532506d06e2e2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=729bd25784924c04aa5f6de9d13a45d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.128022   876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e7f9eb8403e45d88a5532506d06e2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.140235  2069 tablet_bootstrap.cc:492] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:35.143293  2069 tablet_bootstrap.cc:654] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.149859  2070 tablet_bootstrap.cc:492] T 7e7f9eb8403e45d88a5532506d06e2e2 P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:35.152967  2070 tablet_bootstrap.cc:654] T 7e7f9eb8403e45d88a5532506d06e2e2 P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.154242  2071 tablet_bootstrap.cc:492] T 2686a2ecc7594849ba4f0213764f27a8 P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:35.155112  2074 tablet_bootstrap.cc:492] T 7e7f9eb8403e45d88a5532506d06e2e2 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:35.156553  2071 tablet_bootstrap.cc:654] T 2686a2ecc7594849ba4f0213764f27a8 P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.158231  2069 tablet_bootstrap.cc:492] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:35.158373  2068 tablet_bootstrap.cc:492] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:35.158411  2069 ts_tablet_manager.cc:1367] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.018s	user 0.000s	sys 0.008s
I20240503 02:11:35.158754  2073 tablet_bootstrap.cc:492] T 2686a2ecc7594849ba4f0213764f27a8 P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:35.159065  2074 tablet_bootstrap.cc:654] T 7e7f9eb8403e45d88a5532506d06e2e2 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.159242  2069 raft_consensus.cc:348] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:35.159487  2069 raft_consensus.cc:374] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.159579  2069 raft_consensus.cc:729] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.159812  2069 consensus_queue.cc:260] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:35.160177  2068 tablet_bootstrap.cc:654] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.161188  2069 ts_tablet_manager.cc:1398] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.003s
I20240503 02:11:35.162425  2073 tablet_bootstrap.cc:654] T 2686a2ecc7594849ba4f0213764f27a8 P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.163216  2068 tablet_bootstrap.cc:492] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:35.163481  2068 ts_tablet_manager.cc:1367] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:11:35.164682  2068 raft_consensus.cc:348] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.165050  2068 raft_consensus.cc:374] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.165344  2068 raft_consensus.cc:729] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.165720  2068 consensus_queue.cc:260] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.166596  2068 ts_tablet_manager.cc:1398] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:35.167027  2070 tablet_bootstrap.cc:492] T 7e7f9eb8403e45d88a5532506d06e2e2 P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:35.167193  2070 ts_tablet_manager.cc:1367] T 7e7f9eb8403e45d88a5532506d06e2e2 P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.000s
I20240503 02:11:35.168040  2068 tablet_bootstrap.cc:492] T 3c9396ad859e4ea08a06ef08570ac3fb P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:35.168025  2070 raft_consensus.cc:348] T 7e7f9eb8403e45d88a5532506d06e2e2 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:35.168239  2070 raft_consensus.cc:374] T 7e7f9eb8403e45d88a5532506d06e2e2 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.168345  2070 raft_consensus.cc:729] T 7e7f9eb8403e45d88a5532506d06e2e2 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.168630  2070 consensus_queue.cc:260] T 7e7f9eb8403e45d88a5532506d06e2e2 P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:35.170136  2070 ts_tablet_manager.cc:1398] T 7e7f9eb8403e45d88a5532506d06e2e2 P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:35.171432  2074 tablet_bootstrap.cc:492] T 7e7f9eb8403e45d88a5532506d06e2e2 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:35.171656  2074 ts_tablet_manager.cc:1367] T 7e7f9eb8403e45d88a5532506d06e2e2 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.017s	user 0.006s	sys 0.000s
I20240503 02:11:35.172032  2068 tablet_bootstrap.cc:654] T 3c9396ad859e4ea08a06ef08570ac3fb P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.172645  2074 raft_consensus.cc:348] T 7e7f9eb8403e45d88a5532506d06e2e2 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:35.173002  2074 raft_consensus.cc:374] T 7e7f9eb8403e45d88a5532506d06e2e2 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.173154  2074 raft_consensus.cc:729] T 7e7f9eb8403e45d88a5532506d06e2e2 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.173527  2074 consensus_queue.cc:260] T 7e7f9eb8403e45d88a5532506d06e2e2 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:35.176743  2074 ts_tablet_manager.cc:1398] T 7e7f9eb8403e45d88a5532506d06e2e2 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:11:35.180728  2071 tablet_bootstrap.cc:492] T 2686a2ecc7594849ba4f0213764f27a8 P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:35.180728  2073 tablet_bootstrap.cc:492] T 2686a2ecc7594849ba4f0213764f27a8 P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:35.180915  2071 ts_tablet_manager.cc:1367] T 2686a2ecc7594849ba4f0213764f27a8 P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.027s	user 0.011s	sys 0.011s
I20240503 02:11:35.180914  2073 ts_tablet_manager.cc:1367] T 2686a2ecc7594849ba4f0213764f27a8 P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.022s	user 0.002s	sys 0.002s
I20240503 02:11:35.181730  2073 raft_consensus.cc:348] T 2686a2ecc7594849ba4f0213764f27a8 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.181730  2071 raft_consensus.cc:348] T 2686a2ecc7594849ba4f0213764f27a8 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.181951  2073 raft_consensus.cc:374] T 2686a2ecc7594849ba4f0213764f27a8 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.181951  2071 raft_consensus.cc:374] T 2686a2ecc7594849ba4f0213764f27a8 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.182041  2071 raft_consensus.cc:729] T 2686a2ecc7594849ba4f0213764f27a8 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.182041  2073 raft_consensus.cc:729] T 2686a2ecc7594849ba4f0213764f27a8 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.182191  2068 tablet_bootstrap.cc:492] T 3c9396ad859e4ea08a06ef08570ac3fb P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:35.182372  2068 ts_tablet_manager.cc:1367] T 3c9396ad859e4ea08a06ef08570ac3fb P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.014s	user 0.003s	sys 0.007s
I20240503 02:11:35.182260  2071 consensus_queue.cc:260] T 2686a2ecc7594849ba4f0213764f27a8 P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.182297  2073 consensus_queue.cc:260] T 2686a2ecc7594849ba4f0213764f27a8 P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.182883  2071 ts_tablet_manager.cc:1398] T 2686a2ecc7594849ba4f0213764f27a8 P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.002s
I20240503 02:11:35.183265  2071 tablet_bootstrap.cc:492] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:35.183203  2068 raft_consensus.cc:348] T 3c9396ad859e4ea08a06ef08570ac3fb P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.183409  2068 raft_consensus.cc:374] T 3c9396ad859e4ea08a06ef08570ac3fb P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.184684  2073 ts_tablet_manager.cc:1398] T 2686a2ecc7594849ba4f0213764f27a8 P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240503 02:11:35.184933  2071 tablet_bootstrap.cc:654] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.184995  2073 tablet_bootstrap.cc:492] T 3c9396ad859e4ea08a06ef08570ac3fb P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:35.183482  2068 raft_consensus.cc:729] T 3c9396ad859e4ea08a06ef08570ac3fb P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.185416  2068 consensus_queue.cc:260] T 3c9396ad859e4ea08a06ef08570ac3fb P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.186631  2068 ts_tablet_manager.cc:1398] T 3c9396ad859e4ea08a06ef08570ac3fb P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240503 02:11:35.190366  2073 tablet_bootstrap.cc:654] T 3c9396ad859e4ea08a06ef08570ac3fb P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.191607  2071 tablet_bootstrap.cc:492] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:35.191792  2071 ts_tablet_manager.cc:1367] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.005s
I20240503 02:11:35.192512  2073 tablet_bootstrap.cc:492] T 3c9396ad859e4ea08a06ef08570ac3fb P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:35.192679  2073 ts_tablet_manager.cc:1367] T 3c9396ad859e4ea08a06ef08570ac3fb P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.001s
I20240503 02:11:35.192641  2071 raft_consensus.cc:348] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.192899  2071 raft_consensus.cc:374] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.192979  2071 raft_consensus.cc:729] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.193225  2071 consensus_queue.cc:260] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.193444  2073 raft_consensus.cc:348] T 3c9396ad859e4ea08a06ef08570ac3fb P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.193650  2073 raft_consensus.cc:374] T 3c9396ad859e4ea08a06ef08570ac3fb P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.193722  2073 raft_consensus.cc:729] T 3c9396ad859e4ea08a06ef08570ac3fb P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.193868  2071 ts_tablet_manager.cc:1398] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:11:35.193967  2073 consensus_queue.cc:260] T 3c9396ad859e4ea08a06ef08570ac3fb P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.194490  2073 ts_tablet_manager.cc:1398] T 3c9396ad859e4ea08a06ef08570ac3fb P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:11:35.234948  1830 raft_consensus.cc:482] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:35.235158  1830 raft_consensus.cc:504] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.235930  1830 leader_election.cc:288] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:35.236557  1026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686a2ecc7594849ba4f0213764f27a8" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f" is_pre_election: true
I20240503 02:11:35.236557  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686a2ecc7594849ba4f0213764f27a8" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1" is_pre_election: true
I20240503 02:11:35.236794  1289 raft_consensus.cc:2454] T 2686a2ecc7594849ba4f0213764f27a8 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 0.
I20240503 02:11:35.236794  1026 raft_consensus.cc:2454] T 2686a2ecc7594849ba4f0213764f27a8 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 0.
I20240503 02:11:35.237210   699 leader_election.cc:302] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, e8d9a43983f2478ea7ababa09b230177; no voters: 
I20240503 02:11:35.237509  1830 raft_consensus.cc:2781] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:35.237614  1830 raft_consensus.cc:482] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:35.237704  1830 raft_consensus.cc:3037] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.239679  1830 raft_consensus.cc:504] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.240221  1830 leader_election.cc:288] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 election: Requested vote from peers 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:35.240921  1026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686a2ecc7594849ba4f0213764f27a8" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f"
I20240503 02:11:35.240952  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686a2ecc7594849ba4f0213764f27a8" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1"
I20240503 02:11:35.241111  1026 raft_consensus.cc:3037] T 2686a2ecc7594849ba4f0213764f27a8 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.241189  1289 raft_consensus.cc:3037] T 2686a2ecc7594849ba4f0213764f27a8 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.243311  1026 raft_consensus.cc:2454] T 2686a2ecc7594849ba4f0213764f27a8 P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 1.
I20240503 02:11:35.243499  1289 raft_consensus.cc:2454] T 2686a2ecc7594849ba4f0213764f27a8 P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 1.
I20240503 02:11:35.243794   698 leader_election.cc:302] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8d9a43983f2478ea7ababa09b230177, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:35.244122  1830 raft_consensus.cc:2781] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:35.244558  1830 raft_consensus.cc:686] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [term 1 LEADER]: Becoming Leader. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Running, Role: LEADER
I20240503 02:11:35.244928  1830 consensus_queue.cc:237] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.247932   618 catalog_manager.cc:5484] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8d9a43983f2478ea7ababa09b230177 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "e8d9a43983f2478ea7ababa09b230177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:35.352488  1999 raft_consensus.cc:482] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:35.352684  1999 raft_consensus.cc:504] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:35.353312  1999 leader_election.cc:288] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:35.353730   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7f9eb8403e45d88a5532506d06e2e2" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" is_pre_election: true
I20240503 02:11:35.353863  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7f9eb8403e45d88a5532506d06e2e2" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:35.353979   895 raft_consensus.cc:2454] T 7e7f9eb8403e45d88a5532506d06e2e2 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:35.354115  1158 raft_consensus.cc:2454] T 7e7f9eb8403e45d88a5532506d06e2e2 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:35.354422  1355 leader_election.cc:302] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:35.354734  1999 raft_consensus.cc:2781] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:35.354866  1999 raft_consensus.cc:482] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:35.354960  1999 raft_consensus.cc:3037] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.356716  1999 raft_consensus.cc:504] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:35.357254  1999 leader_election.cc:288] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Requested vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:35.357635   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7f9eb8403e45d88a5532506d06e2e2" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
I20240503 02:11:35.357743  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7f9eb8403e45d88a5532506d06e2e2" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:35.357834   895 raft_consensus.cc:3037] T 7e7f9eb8403e45d88a5532506d06e2e2 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.357931  1158 raft_consensus.cc:3037] T 7e7f9eb8403e45d88a5532506d06e2e2 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.359769   895 raft_consensus.cc:2454] T 7e7f9eb8403e45d88a5532506d06e2e2 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:35.359848  1158 raft_consensus.cc:2454] T 7e7f9eb8403e45d88a5532506d06e2e2 P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:35.360162  1355 leader_election.cc:302] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:35.360476  1999 raft_consensus.cc:2781] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:35.360630  1999 raft_consensus.cc:686] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [term 1 LEADER]: Becoming Leader. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Running, Role: LEADER
I20240503 02:11:35.360904  1999 consensus_queue.cc:237] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:35.363572   618 catalog_manager.cc:5484] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6). New cstate: current_term: 1 leader_uuid: "4fda3b525c234f8f88aaefd355e7274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:35.432013  1677 raft_consensus.cc:482] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:35.432257  1677 raft_consensus.cc:504] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.432960  1677 leader_election.cc:288] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189), e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401)
I20240503 02:11:35.433401  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9396ad859e4ea08a06ef08570ac3fb" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1" is_pre_election: true
I20240503 02:11:35.433456   763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9396ad859e4ea08a06ef08570ac3fb" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177" is_pre_election: true
I20240503 02:11:35.433648  1289 raft_consensus.cc:2454] T 3c9396ad859e4ea08a06ef08570ac3fb P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 0.
I20240503 02:11:35.433687   763 raft_consensus.cc:2454] T 3c9396ad859e4ea08a06ef08570ac3fb P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 0.
I20240503 02:11:35.434059   962 leader_election.cc:302] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:35.434355  1677 raft_consensus.cc:2781] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:35.434466  1677 raft_consensus.cc:482] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:35.434621  1677 raft_consensus.cc:3037] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.436622  1677 raft_consensus.cc:504] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.437209  1677 leader_election.cc:288] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 election: Requested vote from peers 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189), e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401)
I20240503 02:11:35.437587  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9396ad859e4ea08a06ef08570ac3fb" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1"
I20240503 02:11:35.437731   763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9396ad859e4ea08a06ef08570ac3fb" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177"
I20240503 02:11:35.437824  1289 raft_consensus.cc:3037] T 3c9396ad859e4ea08a06ef08570ac3fb P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.437944   763 raft_consensus.cc:3037] T 3c9396ad859e4ea08a06ef08570ac3fb P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.440817  1289 raft_consensus.cc:2454] T 3c9396ad859e4ea08a06ef08570ac3fb P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 1.
I20240503 02:11:35.440840   763 raft_consensus.cc:2454] T 3c9396ad859e4ea08a06ef08570ac3fb P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 1.
I20240503 02:11:35.441324   961 leader_election.cc:302] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8d9a43983f2478ea7ababa09b230177, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:35.441627  1677 raft_consensus.cc:2781] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:35.442039  1677 raft_consensus.cc:686] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [term 1 LEADER]: Becoming Leader. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Running, Role: LEADER
I20240503 02:11:35.442304  1677 consensus_queue.cc:237] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:35.445158   618 catalog_manager.cc:5484] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f reported cstate change: term changed from 0 to 1, leader changed from <none> to f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3). New cstate: current_term: 1 leader_uuid: "f5e38dcb24084a1e969f83bf34aef88f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:35.713259  1830 consensus_queue.cc:1035] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:35.719045  2083 consensus_queue.cc:1035] T 2686a2ecc7594849ba4f0213764f27a8 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:35.854224  2088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:35.854640  2088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:11:35.856133  1999 consensus_queue.cc:1035] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:35.860968  1999 consensus_queue.cc:1035] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:35.867053  2088 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 02:11:35.939970  1677 consensus_queue.cc:1035] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:35.982547  1677 consensus_queue.cc:1035] T 3c9396ad859e4ea08a06ef08570ac3fb P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 303.473 ms
  time per row: 0.151737 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240503 02:11:37.282763   614 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44878:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:37.292280  1399 tablet_service.cc:1467] Processing CreateTablet for tablet 4775eb444281418e830ed6944f5f8ec3 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4a7f1413efb443d7a3e6c585336cddc5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.292747  1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4775eb444281418e830ed6944f5f8ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.292771   876 tablet_service.cc:1467] Processing CreateTablet for tablet 4775eb444281418e830ed6944f5f8ec3 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4a7f1413efb443d7a3e6c585336cddc5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.293169   876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4775eb444281418e830ed6944f5f8ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.293464   744 tablet_service.cc:1467] Processing CreateTablet for tablet 9228c7369b9f4ddfb7a2bbf64b2afb02 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4a7f1413efb443d7a3e6c585336cddc5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.293774   744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9228c7369b9f4ddfb7a2bbf64b2afb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.295282  1268 tablet_service.cc:1467] Processing CreateTablet for tablet 9228c7369b9f4ddfb7a2bbf64b2afb02 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4a7f1413efb443d7a3e6c585336cddc5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.295575  1137 tablet_service.cc:1467] Processing CreateTablet for tablet 4775eb444281418e830ed6944f5f8ec3 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4a7f1413efb443d7a3e6c585336cddc5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.295709  1268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9228c7369b9f4ddfb7a2bbf64b2afb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.295719  1138 tablet_service.cc:1467] Processing CreateTablet for tablet 131e73c6862d4f249f8c9f9bafdcb979 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4a7f1413efb443d7a3e6c585336cddc5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.295980  1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4775eb444281418e830ed6944f5f8ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.296974  1400 tablet_service.cc:1467] Processing CreateTablet for tablet 131e73c6862d4f249f8c9f9bafdcb979 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4a7f1413efb443d7a3e6c585336cddc5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.297359  1400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 131e73c6862d4f249f8c9f9bafdcb979. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.298975  1138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 131e73c6862d4f249f8c9f9bafdcb979. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.300686   875 tablet_service.cc:1467] Processing CreateTablet for tablet 131e73c6862d4f249f8c9f9bafdcb979 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4a7f1413efb443d7a3e6c585336cddc5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.301122   875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 131e73c6862d4f249f8c9f9bafdcb979. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.310829  2160 tablet_bootstrap.cc:492] T 4775eb444281418e830ed6944f5f8ec3 P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:37.312917  2160 tablet_bootstrap.cc:654] T 4775eb444281418e830ed6944f5f8ec3 P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.315405  1007 tablet_service.cc:1467] Processing CreateTablet for tablet 9228c7369b9f4ddfb7a2bbf64b2afb02 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4a7f1413efb443d7a3e6c585336cddc5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.315833  1007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9228c7369b9f4ddfb7a2bbf64b2afb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.322782  2163 tablet_bootstrap.cc:492] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:37.323838  2165 tablet_bootstrap.cc:492] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:37.325275  2163 tablet_bootstrap.cc:654] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.325307  2162 tablet_bootstrap.cc:492] T 4775eb444281418e830ed6944f5f8ec3 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:37.331326  2166 tablet_bootstrap.cc:492] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:37.331581  2165 tablet_bootstrap.cc:654] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.332752  2160 tablet_bootstrap.cc:492] T 4775eb444281418e830ed6944f5f8ec3 P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:37.332998  2160 ts_tablet_manager.cc:1367] T 4775eb444281418e830ed6944f5f8ec3 P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.022s	user 0.003s	sys 0.007s
I20240503 02:11:37.333914  2160 raft_consensus.cc:348] T 4775eb444281418e830ed6944f5f8ec3 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:37.334303  2160 raft_consensus.cc:374] T 4775eb444281418e830ed6944f5f8ec3 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.334499  2162 tablet_bootstrap.cc:654] T 4775eb444281418e830ed6944f5f8ec3 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.334502  2160 raft_consensus.cc:729] T 4775eb444281418e830ed6944f5f8ec3 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.334837  2160 consensus_queue.cc:260] T 4775eb444281418e830ed6944f5f8ec3 P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:37.336416  2160 ts_tablet_manager.cc:1398] T 4775eb444281418e830ed6944f5f8ec3 P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:37.336758  2160 tablet_bootstrap.cc:492] T 131e73c6862d4f249f8c9f9bafdcb979 P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:37.338320  2166 tablet_bootstrap.cc:654] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.339334  2160 tablet_bootstrap.cc:654] T 131e73c6862d4f249f8c9f9bafdcb979 P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.341745  2165 tablet_bootstrap.cc:492] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:37.341749  2163 tablet_bootstrap.cc:492] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:37.341917  2163 ts_tablet_manager.cc:1367] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.020s	user 0.012s	sys 0.004s
I20240503 02:11:37.341974  2165 ts_tablet_manager.cc:1367] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.018s	user 0.002s	sys 0.005s
I20240503 02:11:37.342803  2165 raft_consensus.cc:348] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:37.342792  2163 raft_consensus.cc:348] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:37.343171  2165 raft_consensus.cc:374] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.343191  2163 raft_consensus.cc:374] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.343299  2163 raft_consensus.cc:729] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.343465  2165 raft_consensus.cc:729] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.343616  2163 consensus_queue.cc:260] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:37.344784  2163 ts_tablet_manager.cc:1398] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:37.345114  2163 tablet_bootstrap.cc:492] T 131e73c6862d4f249f8c9f9bafdcb979 P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:37.345165  2159 tablet_bootstrap.cc:492] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:37.346277  2166 tablet_bootstrap.cc:492] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:37.346581  2166 ts_tablet_manager.cc:1367] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.016s	user 0.002s	sys 0.003s
I20240503 02:11:37.346853  2163 tablet_bootstrap.cc:654] T 131e73c6862d4f249f8c9f9bafdcb979 P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.343966  2165 consensus_queue.cc:260] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:37.349599  2165 ts_tablet_manager.cc:1398] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240503 02:11:37.350066  2160 tablet_bootstrap.cc:492] T 131e73c6862d4f249f8c9f9bafdcb979 P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:37.350322  2160 ts_tablet_manager.cc:1367] T 131e73c6862d4f249f8c9f9bafdcb979 P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.004s
I20240503 02:11:37.351282  2160 raft_consensus.cc:348] T 131e73c6862d4f249f8c9f9bafdcb979 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:37.351584  2160 raft_consensus.cc:374] T 131e73c6862d4f249f8c9f9bafdcb979 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.351707  2160 raft_consensus.cc:729] T 131e73c6862d4f249f8c9f9bafdcb979 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.350783  2166 raft_consensus.cc:348] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:37.352089  2160 consensus_queue.cc:260] T 131e73c6862d4f249f8c9f9bafdcb979 P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:37.353514  2160 ts_tablet_manager.cc:1398] T 131e73c6862d4f249f8c9f9bafdcb979 P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:37.353871  2166 raft_consensus.cc:374] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.354391  2166 raft_consensus.cc:729] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.355438  2162 tablet_bootstrap.cc:492] T 4775eb444281418e830ed6944f5f8ec3 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:37.355451  2163 tablet_bootstrap.cc:492] T 131e73c6862d4f249f8c9f9bafdcb979 P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:37.355620  2163 ts_tablet_manager.cc:1367] T 131e73c6862d4f249f8c9f9bafdcb979 P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.011s	user 0.008s	sys 0.000s
I20240503 02:11:37.355662  2162 ts_tablet_manager.cc:1367] T 4775eb444281418e830ed6944f5f8ec3 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.031s	user 0.003s	sys 0.003s
I20240503 02:11:37.356343  2163 raft_consensus.cc:348] T 131e73c6862d4f249f8c9f9bafdcb979 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:37.356523  2163 raft_consensus.cc:374] T 131e73c6862d4f249f8c9f9bafdcb979 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.356609  2163 raft_consensus.cc:729] T 131e73c6862d4f249f8c9f9bafdcb979 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.356696  2162 raft_consensus.cc:348] T 4775eb444281418e830ed6944f5f8ec3 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:37.356936  2162 raft_consensus.cc:374] T 4775eb444281418e830ed6944f5f8ec3 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.356878  2163 consensus_queue.cc:260] T 131e73c6862d4f249f8c9f9bafdcb979 P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:37.357033  2162 raft_consensus.cc:729] T 4775eb444281418e830ed6944f5f8ec3 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.357259  2162 consensus_queue.cc:260] T 4775eb444281418e830ed6944f5f8ec3 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:37.357522  2163 ts_tablet_manager.cc:1398] T 131e73c6862d4f249f8c9f9bafdcb979 P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:11:37.358657  2162 ts_tablet_manager.cc:1398] T 4775eb444281418e830ed6944f5f8ec3 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 02:11:37.358994  2162 tablet_bootstrap.cc:492] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:37.359408  2166 consensus_queue.cc:260] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:37.360066  2166 ts_tablet_manager.cc:1398] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.012s	user 0.001s	sys 0.001s
I20240503 02:11:37.360474  2159 tablet_bootstrap.cc:654] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.363309  2162 tablet_bootstrap.cc:654] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.365717  2162 tablet_bootstrap.cc:492] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:37.365820  2159 tablet_bootstrap.cc:492] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:37.365885  2162 ts_tablet_manager.cc:1367] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.003s
I20240503 02:11:37.366003  2159 ts_tablet_manager.cc:1367] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.021s	user 0.003s	sys 0.003s
I20240503 02:11:37.366640  2162 raft_consensus.cc:348] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:37.366863  2162 raft_consensus.cc:374] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.366818  2159 raft_consensus.cc:348] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:37.366954  2162 raft_consensus.cc:729] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.367036  2159 raft_consensus.cc:374] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.367131  2159 raft_consensus.cc:729] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.367210  2162 consensus_queue.cc:260] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:37.367354  2159 consensus_queue.cc:260] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:37.367712  2162 ts_tablet_manager.cc:1398] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 02:11:37.367875  2159 ts_tablet_manager.cc:1398] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:11:37.415282  2124 raft_consensus.cc:482] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:37.415460  2124 raft_consensus.cc:504] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:37.416079  2124 leader_election.cc:288] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401), 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189)
I20240503 02:11:37.416574  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9228c7369b9f4ddfb7a2bbf64b2afb02" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1" is_pre_election: true
I20240503 02:11:37.416589   763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9228c7369b9f4ddfb7a2bbf64b2afb02" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177" is_pre_election: true
I20240503 02:11:37.416772  1289 raft_consensus.cc:2454] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 0.
I20240503 02:11:37.416832   763 raft_consensus.cc:2454] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 0.
I20240503 02:11:37.417239   962 leader_election.cc:302] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:37.417519  2124 raft_consensus.cc:2781] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:37.417634  2124 raft_consensus.cc:482] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:37.417733  2124 raft_consensus.cc:3037] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.419808  2124 raft_consensus.cc:504] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:37.420466  2124 leader_election.cc:288] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 election: Requested vote from peers e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401), 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189)
I20240503 02:11:37.420905   763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9228c7369b9f4ddfb7a2bbf64b2afb02" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177"
I20240503 02:11:37.420991  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9228c7369b9f4ddfb7a2bbf64b2afb02" candidate_uuid: "f5e38dcb24084a1e969f83bf34aef88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1"
I20240503 02:11:37.421123   763 raft_consensus.cc:3037] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.421218  1289 raft_consensus.cc:3037] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.423103   763 raft_consensus.cc:2454] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 1.
I20240503 02:11:37.423209  1289 raft_consensus.cc:2454] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5e38dcb24084a1e969f83bf34aef88f in term 1.
I20240503 02:11:37.423496   961 leader_election.cc:302] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8d9a43983f2478ea7ababa09b230177, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:37.423759  2124 raft_consensus.cc:2781] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:37.424172  2124 raft_consensus.cc:686] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [term 1 LEADER]: Becoming Leader. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Running, Role: LEADER
I20240503 02:11:37.424436  2124 consensus_queue.cc:237] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:37.427479   614 catalog_manager.cc:5484] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f reported cstate change: term changed from 0 to 1, leader changed from <none> to f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3). New cstate: current_term: 1 leader_uuid: "f5e38dcb24084a1e969f83bf34aef88f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:37.432698  2161 raft_consensus.cc:482] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:37.432852  2161 raft_consensus.cc:504] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:37.433562  2161 leader_election.cc:288] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:37.433950  1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131e73c6862d4f249f8c9f9bafdcb979" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e" is_pre_election: true
I20240503 02:11:37.434023  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131e73c6862d4f249f8c9f9bafdcb979" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:37.434190  1420 raft_consensus.cc:2454] T 131e73c6862d4f249f8c9f9bafdcb979 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 0.
I20240503 02:11:37.434271  1158 raft_consensus.cc:2454] T 131e73c6862d4f249f8c9f9bafdcb979 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 0.
I20240503 02:11:37.434773   831 leader_election.cc:302] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:37.435043  2161 raft_consensus.cc:2781] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:37.435168  2161 raft_consensus.cc:482] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:37.435263  2161 raft_consensus.cc:3037] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.437288  2161 raft_consensus.cc:504] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:37.437762  2161 leader_election.cc:288] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 election: Requested vote from peers 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:37.438211  1420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131e73c6862d4f249f8c9f9bafdcb979" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e"
I20240503 02:11:37.438282  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "131e73c6862d4f249f8c9f9bafdcb979" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:37.438436  1420 raft_consensus.cc:3037] T 131e73c6862d4f249f8c9f9bafdcb979 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.438513  1158 raft_consensus.cc:3037] T 131e73c6862d4f249f8c9f9bafdcb979 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.441311  1420 raft_consensus.cc:2454] T 131e73c6862d4f249f8c9f9bafdcb979 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 1.
I20240503 02:11:37.441296  1158 raft_consensus.cc:2454] T 131e73c6862d4f249f8c9f9bafdcb979 P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 1.
I20240503 02:11:37.441756   831 leader_election.cc:302] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:37.442039  2161 raft_consensus.cc:2781] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:37.442196  2161 raft_consensus.cc:686] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 LEADER]: Becoming Leader. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Running, Role: LEADER
I20240503 02:11:37.442495  2161 consensus_queue.cc:237] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:37.445369   614 catalog_manager.cc:5484] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2). New cstate: current_term: 1 leader_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:37.617794  1999 raft_consensus.cc:482] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:37.618021  1999 raft_consensus.cc:504] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:37.618701  1999 leader_election.cc:288] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:37.619323   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4775eb444281418e830ed6944f5f8ec3" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" is_pre_election: true
I20240503 02:11:37.619343  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4775eb444281418e830ed6944f5f8ec3" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:37.619583  1158 raft_consensus.cc:2454] T 4775eb444281418e830ed6944f5f8ec3 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:37.619604   895 raft_consensus.cc:2454] T 4775eb444281418e830ed6944f5f8ec3 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:37.620028  1355 leader_election.cc:302] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 4fda3b525c234f8f88aaefd355e7274e; no voters: 
I20240503 02:11:37.620358  1999 raft_consensus.cc:2781] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:37.620455  1999 raft_consensus.cc:482] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:37.620571  1999 raft_consensus.cc:3037] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.622776  1999 raft_consensus.cc:504] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:37.623382  1999 leader_election.cc:288] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Requested vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:37.623863   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4775eb444281418e830ed6944f5f8ec3" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
I20240503 02:11:37.624092   895 raft_consensus.cc:3037] T 4775eb444281418e830ed6944f5f8ec3 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.624037  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4775eb444281418e830ed6944f5f8ec3" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:37.624243  1158 raft_consensus.cc:3037] T 4775eb444281418e830ed6944f5f8ec3 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.626111   895 raft_consensus.cc:2454] T 4775eb444281418e830ed6944f5f8ec3 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:37.626186  1158 raft_consensus.cc:2454] T 4775eb444281418e830ed6944f5f8ec3 P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:37.626624  1355 leader_election.cc:302] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:37.626974  2178 raft_consensus.cc:2781] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:37.627135  2178 raft_consensus.cc:686] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [term 1 LEADER]: Becoming Leader. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Running, Role: LEADER
I20240503 02:11:37.627471  2178 consensus_queue.cc:237] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:37.630342   614 catalog_manager.cc:5484] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6). New cstate: current_term: 1 leader_uuid: "4fda3b525c234f8f88aaefd355e7274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:37.908228  2161 consensus_queue.cc:1035] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:37.920475  2161 consensus_queue.cc:1035] T 131e73c6862d4f249f8c9f9bafdcb979 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:37.956779  2124 consensus_queue.cc:1035] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:37.968453  2124 consensus_queue.cc:1035] T 9228c7369b9f4ddfb7a2bbf64b2afb02 P f5e38dcb24084a1e969f83bf34aef88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:38.064051  2180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:38.064517  2180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:38.075101  2180 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 02:11:38.153878  2178 consensus_queue.cc:1035] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:38.177419  2221 consensus_queue.cc:1035] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 293.62 ms
  time per row: 0.14681 ms
I20240503 02:11:38.531531   621 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44878:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:38.541391   744 tablet_service.cc:1467] Processing CreateTablet for tablet d320d692be6c47b6b9ec43fe3dec719c (DEFAULT_TABLE table=rebalance_test_table_8 [id=d1c6c615caf44e16a1216813b9bec46f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:38.541846   744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d320d692be6c47b6b9ec43fe3dec719c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:38.541942  1007 tablet_service.cc:1467] Processing CreateTablet for tablet d320d692be6c47b6b9ec43fe3dec719c (DEFAULT_TABLE table=rebalance_test_table_8 [id=d1c6c615caf44e16a1216813b9bec46f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:38.542387  1007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d320d692be6c47b6b9ec43fe3dec719c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:38.543290  1138 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1a504c6d8e4bf89cc813b9cedbccd6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d1c6c615caf44e16a1216813b9bec46f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:38.543733  1138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1a504c6d8e4bf89cc813b9cedbccd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:38.543819   876 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1a504c6d8e4bf89cc813b9cedbccd6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d1c6c615caf44e16a1216813b9bec46f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:38.544255   876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1a504c6d8e4bf89cc813b9cedbccd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:38.546083  1006 tablet_service.cc:1467] Processing CreateTablet for tablet c4ed1defa1314b86a5186c3349169694 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d1c6c615caf44e16a1216813b9bec46f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:38.546459  1006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ed1defa1314b86a5186c3349169694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:38.547468  1268 tablet_service.cc:1467] Processing CreateTablet for tablet d320d692be6c47b6b9ec43fe3dec719c (DEFAULT_TABLE table=rebalance_test_table_8 [id=d1c6c615caf44e16a1216813b9bec46f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:38.547880  1268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d320d692be6c47b6b9ec43fe3dec719c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:38.551043  1269 tablet_service.cc:1467] Processing CreateTablet for tablet c4ed1defa1314b86a5186c3349169694 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d1c6c615caf44e16a1216813b9bec46f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:38.551461  1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ed1defa1314b86a5186c3349169694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:38.555138   743 tablet_service.cc:1467] Processing CreateTablet for tablet c4ed1defa1314b86a5186c3349169694 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d1c6c615caf44e16a1216813b9bec46f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:38.555617   743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ed1defa1314b86a5186c3349169694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:38.559592  2254 tablet_bootstrap.cc:492] T d320d692be6c47b6b9ec43fe3dec719c P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:38.559814  1399 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1a504c6d8e4bf89cc813b9cedbccd6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d1c6c615caf44e16a1216813b9bec46f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:38.560266  1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1a504c6d8e4bf89cc813b9cedbccd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:38.562193  2254 tablet_bootstrap.cc:654] T d320d692be6c47b6b9ec43fe3dec719c P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:38.575371  2256 tablet_bootstrap.cc:492] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:38.577365  2256 tablet_bootstrap.cc:654] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:38.596243  2256 tablet_bootstrap.cc:492] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:38.596470  2256 ts_tablet_manager.cc:1367] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.012s
I20240503 02:11:38.598851  2256 raft_consensus.cc:348] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:38.599331  2256 raft_consensus.cc:374] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:38.599491  2256 raft_consensus.cc:729] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:38.599872  2256 consensus_queue.cc:260] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:38.601935  2256 ts_tablet_manager.cc:1398] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 02:11:38.602329  2256 tablet_bootstrap.cc:492] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:38.604635  2256 tablet_bootstrap.cc:654] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:38.612473  2258 tablet_bootstrap.cc:492] T d320d692be6c47b6b9ec43fe3dec719c P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:38.616129  2258 tablet_bootstrap.cc:654] T d320d692be6c47b6b9ec43fe3dec719c P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:38.617883  2260 tablet_bootstrap.cc:492] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:38.619984  2260 tablet_bootstrap.cc:654] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:38.621716  2259 tablet_bootstrap.cc:492] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:38.623668  2259 tablet_bootstrap.cc:654] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:38.626449  2253 tablet_bootstrap.cc:492] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:38.627981  2258 tablet_bootstrap.cc:492] T d320d692be6c47b6b9ec43fe3dec719c P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:38.627981  2260 tablet_bootstrap.cc:492] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:38.628165  2260 ts_tablet_manager.cc:1367] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.000s
I20240503 02:11:38.628165  2258 ts_tablet_manager.cc:1367] T d320d692be6c47b6b9ec43fe3dec719c P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.016s	user 0.003s	sys 0.000s
I20240503 02:11:38.628294  2253 tablet_bootstrap.cc:654] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:38.629073  2260 raft_consensus.cc:348] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:38.629073  2258 raft_consensus.cc:348] T d320d692be6c47b6b9ec43fe3dec719c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:38.629323  2260 raft_consensus.cc:374] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:38.629323  2258 raft_consensus.cc:374] T d320d692be6c47b6b9ec43fe3dec719c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:38.629433  2260 raft_consensus.cc:729] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:38.629433  2258 raft_consensus.cc:729] T d320d692be6c47b6b9ec43fe3dec719c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:38.629736  2258 consensus_queue.cc:260] T d320d692be6c47b6b9ec43fe3dec719c P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:38.630342  2259 tablet_bootstrap.cc:492] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:38.630442  2258 ts_tablet_manager.cc:1398] T d320d692be6c47b6b9ec43fe3dec719c P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:38.630460  2253 tablet_bootstrap.cc:492] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:38.630559  2259 ts_tablet_manager.cc:1367] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.004s
I20240503 02:11:38.630630  2253 ts_tablet_manager.cc:1367] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240503 02:11:38.630749  2260 consensus_queue.cc:260] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:38.631501  2259 raft_consensus.cc:348] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:38.631693  2260 ts_tablet_manager.cc:1398] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:38.631805  2259 raft_consensus.cc:374] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:38.631930  2259 raft_consensus.cc:729] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:38.632293  2259 consensus_queue.cc:260] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:38.632934  2259 ts_tablet_manager.cc:1398] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:38.633045  2254 tablet_bootstrap.cc:492] T d320d692be6c47b6b9ec43fe3dec719c P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:38.633260  2254 ts_tablet_manager.cc:1367] T d320d692be6c47b6b9ec43fe3dec719c P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.074s	user 0.008s	sys 0.037s
I20240503 02:11:38.634155  2254 raft_consensus.cc:348] T d320d692be6c47b6b9ec43fe3dec719c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:38.634506  2254 raft_consensus.cc:374] T d320d692be6c47b6b9ec43fe3dec719c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:38.634697  2254 raft_consensus.cc:729] T d320d692be6c47b6b9ec43fe3dec719c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:38.634943  2258 tablet_bootstrap.cc:492] T c4ed1defa1314b86a5186c3349169694 P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:38.635087  2254 consensus_queue.cc:260] T d320d692be6c47b6b9ec43fe3dec719c P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:38.635748  2254 ts_tablet_manager.cc:1398] T d320d692be6c47b6b9ec43fe3dec719c P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 02:11:38.637202  2258 tablet_bootstrap.cc:654] T c4ed1defa1314b86a5186c3349169694 P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:38.637370  2254 tablet_bootstrap.cc:492] T c4ed1defa1314b86a5186c3349169694 P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:38.638700  2253 raft_consensus.cc:348] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:38.639214  2256 tablet_bootstrap.cc:492] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:38.639345  2253 raft_consensus.cc:374] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:38.639415  2256 ts_tablet_manager.cc:1367] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.037s	user 0.003s	sys 0.007s
I20240503 02:11:38.639475  2253 raft_consensus.cc:729] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:38.639729  2253 consensus_queue.cc:260] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:38.640229  2254 tablet_bootstrap.cc:654] T c4ed1defa1314b86a5186c3349169694 P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:38.640309  2256 raft_consensus.cc:348] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:38.640550  2256 raft_consensus.cc:374] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:38.640630  2256 raft_consensus.cc:729] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:38.640712  2253 ts_tablet_manager.cc:1398] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:11:38.640909  2256 consensus_queue.cc:260] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:38.641067  2209 raft_consensus.cc:482] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:38.641206  2209 raft_consensus.cc:504] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:38.641552  2256 ts_tablet_manager.cc:1398] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.002s
I20240503 02:11:38.642047  2209 leader_election.cc:288] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511), 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189)
I20240503 02:11:38.642848  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d320d692be6c47b6b9ec43fe3dec719c" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f" is_pre_election: true
I20240503 02:11:38.643070  1027 raft_consensus.cc:2454] T d320d692be6c47b6b9ec43fe3dec719c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 0.
I20240503 02:11:38.643898   698 leader_election.cc:302] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8d9a43983f2478ea7ababa09b230177, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:38.644556  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d320d692be6c47b6b9ec43fe3dec719c" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1" is_pre_election: true
I20240503 02:11:38.644794  1288 raft_consensus.cc:2454] T d320d692be6c47b6b9ec43fe3dec719c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 0.
I20240503 02:11:38.645745  2209 raft_consensus.cc:2781] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:38.645877  2209 raft_consensus.cc:482] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:38.645978  2209 raft_consensus.cc:3037] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.647810  2258 tablet_bootstrap.cc:492] T c4ed1defa1314b86a5186c3349169694 P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:38.648018  2258 ts_tablet_manager.cc:1367] T c4ed1defa1314b86a5186c3349169694 P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.013s	user 0.008s	sys 0.000s
I20240503 02:11:38.648836  2258 raft_consensus.cc:348] T c4ed1defa1314b86a5186c3349169694 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:38.648895  2209 raft_consensus.cc:504] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:38.649034  2254 tablet_bootstrap.cc:492] T c4ed1defa1314b86a5186c3349169694 P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:38.649186  2254 ts_tablet_manager.cc:1367] T c4ed1defa1314b86a5186c3349169694 P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.012s	user 0.000s	sys 0.009s
I20240503 02:11:38.649183  2258 raft_consensus.cc:374] T c4ed1defa1314b86a5186c3349169694 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:38.649271  2258 raft_consensus.cc:729] T c4ed1defa1314b86a5186c3349169694 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:38.649593  2258 consensus_queue.cc:260] T c4ed1defa1314b86a5186c3349169694 P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:38.649994  2254 raft_consensus.cc:348] T c4ed1defa1314b86a5186c3349169694 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:38.650121  2258 ts_tablet_manager.cc:1398] T c4ed1defa1314b86a5186c3349169694 P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:38.650252  2254 raft_consensus.cc:374] T c4ed1defa1314b86a5186c3349169694 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:38.650350  2254 raft_consensus.cc:729] T c4ed1defa1314b86a5186c3349169694 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:38.650648  2254 consensus_queue.cc:260] T c4ed1defa1314b86a5186c3349169694 P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:38.651454  2254 ts_tablet_manager.cc:1398] T c4ed1defa1314b86a5186c3349169694 P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 02:11:38.651930  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d320d692be6c47b6b9ec43fe3dec719c" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f"
I20240503 02:11:38.652120  1027 raft_consensus.cc:3037] T d320d692be6c47b6b9ec43fe3dec719c P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.654918  1027 raft_consensus.cc:2454] T d320d692be6c47b6b9ec43fe3dec719c P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 1.
I20240503 02:11:38.655388  2209 leader_election.cc:288] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 election: Requested vote from peers f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511), 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189)
I20240503 02:11:38.655388   698 leader_election.cc:302] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8d9a43983f2478ea7ababa09b230177, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:38.655762  2209 raft_consensus.cc:2781] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:38.655937  2209 raft_consensus.cc:686] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [term 1 LEADER]: Becoming Leader. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Running, Role: LEADER
I20240503 02:11:38.655959  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d320d692be6c47b6b9ec43fe3dec719c" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1"
I20240503 02:11:38.656172  1288 raft_consensus.cc:3037] T d320d692be6c47b6b9ec43fe3dec719c P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.656312  2209 consensus_queue.cc:237] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:38.658447  1288 raft_consensus.cc:2454] T d320d692be6c47b6b9ec43fe3dec719c P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 1.
I20240503 02:11:38.659461   622 catalog_manager.cc:5484] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8d9a43983f2478ea7ababa09b230177 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "e8d9a43983f2478ea7ababa09b230177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:38.704479  2209 raft_consensus.cc:482] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:38.704720  2209 raft_consensus.cc:504] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:38.705401  2209 leader_election.cc:288] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:38.706033  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ed1defa1314b86a5186c3349169694" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f" is_pre_election: true
I20240503 02:11:38.706035  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ed1defa1314b86a5186c3349169694" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1" is_pre_election: true
I20240503 02:11:38.706270  1027 raft_consensus.cc:2454] T c4ed1defa1314b86a5186c3349169694 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 0.
I20240503 02:11:38.706290  1288 raft_consensus.cc:2454] T c4ed1defa1314b86a5186c3349169694 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 0.
I20240503 02:11:38.706746   698 leader_election.cc:302] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8d9a43983f2478ea7ababa09b230177, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:38.707064  2209 raft_consensus.cc:2781] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:38.707162  2209 raft_consensus.cc:482] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:38.707310  2209 raft_consensus.cc:3037] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.709394  2209 raft_consensus.cc:504] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:38.710003  2209 leader_election.cc:288] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 election: Requested vote from peers 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:38.710532  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ed1defa1314b86a5186c3349169694" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f"
I20240503 02:11:38.710543  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ed1defa1314b86a5186c3349169694" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1"
I20240503 02:11:38.710776  1027 raft_consensus.cc:3037] T c4ed1defa1314b86a5186c3349169694 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.710783  1288 raft_consensus.cc:3037] T c4ed1defa1314b86a5186c3349169694 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.712682  1027 raft_consensus.cc:2454] T c4ed1defa1314b86a5186c3349169694 P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 1.
I20240503 02:11:38.712710  1288 raft_consensus.cc:2454] T c4ed1defa1314b86a5186c3349169694 P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 1.
I20240503 02:11:38.713160   698 leader_election.cc:302] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8d9a43983f2478ea7ababa09b230177, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:38.713518  2209 raft_consensus.cc:2781] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:38.713709  2209 raft_consensus.cc:686] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [term 1 LEADER]: Becoming Leader. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Running, Role: LEADER
I20240503 02:11:38.714010  2209 consensus_queue.cc:237] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:38.717171   622 catalog_manager.cc:5484] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8d9a43983f2478ea7ababa09b230177 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "e8d9a43983f2478ea7ababa09b230177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:38.794528  2007 raft_consensus.cc:482] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:38.794802  2007 raft_consensus.cc:504] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:38.795406  2007 leader_election.cc:288] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773), 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135)
I20240503 02:11:38.796008  1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a1a504c6d8e4bf89cc813b9cedbccd6" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:38.796047  1419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a1a504c6d8e4bf89cc813b9cedbccd6" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e" is_pre_election: true
I20240503 02:11:38.796214  1157 raft_consensus.cc:2454] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 0.
I20240503 02:11:38.796249  1419 raft_consensus.cc:2454] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 0.
I20240503 02:11:38.796692   831 leader_election.cc:302] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:38.796985  2007 raft_consensus.cc:2781] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:38.797098  2007 raft_consensus.cc:482] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:38.797210  2007 raft_consensus.cc:3037] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.799237  2007 raft_consensus.cc:504] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:38.799767  2007 leader_election.cc:288] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 election: Requested vote from peers 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773), 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6:40135)
I20240503 02:11:38.800397  1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a1a504c6d8e4bf89cc813b9cedbccd6" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:38.800405  1419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a1a504c6d8e4bf89cc813b9cedbccd6" candidate_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fda3b525c234f8f88aaefd355e7274e"
I20240503 02:11:38.800613  1419 raft_consensus.cc:3037] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.800613  1157 raft_consensus.cc:3037] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.803543  1419 raft_consensus.cc:2454] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 1.
I20240503 02:11:38.803543  1157 raft_consensus.cc:2454] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88791c3c9c9341a6a2d8a6b5d3aa1f1b in term 1.
I20240503 02:11:38.804015   831 leader_election.cc:302] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:38.804313  2007 raft_consensus.cc:2781] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:38.804478  2007 raft_consensus.cc:686] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 LEADER]: Becoming Leader. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Running, Role: LEADER
I20240503 02:11:38.804719  2007 consensus_queue.cc:237] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:38.807888   622 catalog_manager.cc:5484] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2). New cstate: current_term: 1 leader_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:39.158039  2209 consensus_queue.cc:1035] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:39.166402  2209 consensus_queue.cc:1035] T d320d692be6c47b6b9ec43fe3dec719c P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:39.195751  2144 consensus_queue.cc:1035] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:39.221741  2007 consensus_queue.cc:1035] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:39.228796  2007 consensus_queue.cc:1035] T 9a1a504c6d8e4bf89cc813b9cedbccd6 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:39.232326  2144 consensus_queue.cc:1035] T c4ed1defa1314b86a5186c3349169694 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:39.259666  2271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:39.259989  2271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:39.268815  2271 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'

INSERT report
    rows total: 2000
    time total: 267.956 ms
  time per row: 0.133978 ms
I20240503 02:11:39.692646   618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44878:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:39.702847  1399 tablet_service.cc:1467] Processing CreateTablet for tablet 0751c6ca11054c0f93d45574c49dc1ac (DEFAULT_TABLE table=rebalance_test_table_9 [id=cc1139a09cdf4df59eda164c6169289d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.703367  1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0751c6ca11054c0f93d45574c49dc1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.703401   876 tablet_service.cc:1467] Processing CreateTablet for tablet 0751c6ca11054c0f93d45574c49dc1ac (DEFAULT_TABLE table=rebalance_test_table_9 [id=cc1139a09cdf4df59eda164c6169289d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.703760  1400 tablet_service.cc:1467] Processing CreateTablet for tablet c04e46c4e09740b2a847d140e0a56b12 (DEFAULT_TABLE table=rebalance_test_table_9 [id=cc1139a09cdf4df59eda164c6169289d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.703838   876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0751c6ca11054c0f93d45574c49dc1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.704078  1400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04e46c4e09740b2a847d140e0a56b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.705138  1268 tablet_service.cc:1467] Processing CreateTablet for tablet d68da7e811f043dabeb7645bc3d8c4cd (DEFAULT_TABLE table=rebalance_test_table_9 [id=cc1139a09cdf4df59eda164c6169289d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.705552  1268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68da7e811f043dabeb7645bc3d8c4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.706593  1006 tablet_service.cc:1467] Processing CreateTablet for tablet d68da7e811f043dabeb7645bc3d8c4cd (DEFAULT_TABLE table=rebalance_test_table_9 [id=cc1139a09cdf4df59eda164c6169289d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.707036  1006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68da7e811f043dabeb7645bc3d8c4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.708269   875 tablet_service.cc:1467] Processing CreateTablet for tablet c04e46c4e09740b2a847d140e0a56b12 (DEFAULT_TABLE table=rebalance_test_table_9 [id=cc1139a09cdf4df59eda164c6169289d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.708673   875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04e46c4e09740b2a847d140e0a56b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.711076   743 tablet_service.cc:1467] Processing CreateTablet for tablet d68da7e811f043dabeb7645bc3d8c4cd (DEFAULT_TABLE table=rebalance_test_table_9 [id=cc1139a09cdf4df59eda164c6169289d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.711485   743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68da7e811f043dabeb7645bc3d8c4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.712503  1137 tablet_service.cc:1467] Processing CreateTablet for tablet c04e46c4e09740b2a847d140e0a56b12 (DEFAULT_TABLE table=rebalance_test_table_9 [id=cc1139a09cdf4df59eda164c6169289d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.712960  1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c04e46c4e09740b2a847d140e0a56b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.720002  1138 tablet_service.cc:1467] Processing CreateTablet for tablet 0751c6ca11054c0f93d45574c49dc1ac (DEFAULT_TABLE table=rebalance_test_table_9 [id=cc1139a09cdf4df59eda164c6169289d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.720607  1138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0751c6ca11054c0f93d45574c49dc1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.721884  2339 tablet_bootstrap.cc:492] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:39.723784  2339 tablet_bootstrap.cc:654] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.730001  2340 tablet_bootstrap.cc:492] T 0751c6ca11054c0f93d45574c49dc1ac P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:39.733917  2340 tablet_bootstrap.cc:654] T 0751c6ca11054c0f93d45574c49dc1ac P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.735181  2338 tablet_bootstrap.cc:492] T d68da7e811f043dabeb7645bc3d8c4cd P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:39.735805  2337 tablet_bootstrap.cc:492] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:39.738215  2338 tablet_bootstrap.cc:654] T d68da7e811f043dabeb7645bc3d8c4cd P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.738230  2337 tablet_bootstrap.cc:654] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.739854  2340 tablet_bootstrap.cc:492] T 0751c6ca11054c0f93d45574c49dc1ac P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:39.739847  2339 tablet_bootstrap.cc:492] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:39.740036  2339 ts_tablet_manager.cc:1367] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.005s
I20240503 02:11:39.740098  2340 ts_tablet_manager.cc:1367] T 0751c6ca11054c0f93d45574c49dc1ac P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.000s
I20240503 02:11:39.740759  2343 tablet_bootstrap.cc:492] T d68da7e811f043dabeb7645bc3d8c4cd P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:39.740965  2339 raft_consensus.cc:348] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:39.741187  2339 raft_consensus.cc:374] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.741282  2339 raft_consensus.cc:729] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.741153  2340 raft_consensus.cc:348] T 0751c6ca11054c0f93d45574c49dc1ac P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:39.741421  2340 raft_consensus.cc:374] T 0751c6ca11054c0f93d45574c49dc1ac P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.741521  2340 raft_consensus.cc:729] T 0751c6ca11054c0f93d45574c49dc1ac P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.741521  2339 consensus_queue.cc:260] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:39.742113  2339 ts_tablet_manager.cc:1398] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:11:39.741986  2340 consensus_queue.cc:260] T 0751c6ca11054c0f93d45574c49dc1ac P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:39.744279  2340 ts_tablet_manager.cc:1398] T 0751c6ca11054c0f93d45574c49dc1ac P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:11:39.744649  2340 tablet_bootstrap.cc:492] T c04e46c4e09740b2a847d140e0a56b12 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:39.746224  2342 tablet_bootstrap.cc:492] T c04e46c4e09740b2a847d140e0a56b12 P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:39.746428  2340 tablet_bootstrap.cc:654] T c04e46c4e09740b2a847d140e0a56b12 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.754362  2340 tablet_bootstrap.cc:492] T c04e46c4e09740b2a847d140e0a56b12 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:39.754659  2340 ts_tablet_manager.cc:1367] T c04e46c4e09740b2a847d140e0a56b12 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.010s	user 0.007s	sys 0.000s
I20240503 02:11:39.754889  2338 tablet_bootstrap.cc:492] T d68da7e811f043dabeb7645bc3d8c4cd P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:39.755154  2338 ts_tablet_manager.cc:1367] T d68da7e811f043dabeb7645bc3d8c4cd P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.020s	user 0.006s	sys 0.006s
I20240503 02:11:39.755563  2340 raft_consensus.cc:348] T c04e46c4e09740b2a847d140e0a56b12 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:39.755841  2340 raft_consensus.cc:374] T c04e46c4e09740b2a847d140e0a56b12 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.755944  2340 raft_consensus.cc:729] T c04e46c4e09740b2a847d140e0a56b12 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.756232  2337 tablet_bootstrap.cc:492] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:39.756239  2338 raft_consensus.cc:348] T d68da7e811f043dabeb7645bc3d8c4cd P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:39.756440  2337 ts_tablet_manager.cc:1367] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.021s	user 0.003s	sys 0.007s
I20240503 02:11:39.756346  2340 consensus_queue.cc:260] T c04e46c4e09740b2a847d140e0a56b12 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:39.756561  2338 raft_consensus.cc:374] T d68da7e811f043dabeb7645bc3d8c4cd P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.756702  2338 raft_consensus.cc:729] T d68da7e811f043dabeb7645bc3d8c4cd P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.756939  2340 ts_tablet_manager.cc:1398] T c04e46c4e09740b2a847d140e0a56b12 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:11:39.756976  2338 consensus_queue.cc:260] T d68da7e811f043dabeb7645bc3d8c4cd P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:39.757565  2338 ts_tablet_manager.cc:1398] T d68da7e811f043dabeb7645bc3d8c4cd P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:11:39.759460  2337 raft_consensus.cc:348] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:39.759773  2337 raft_consensus.cc:374] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.759920  2337 raft_consensus.cc:729] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.760151  2337 consensus_queue.cc:260] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:39.760654  2337 ts_tablet_manager.cc:1398] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.001s
I20240503 02:11:39.761057  2337 tablet_bootstrap.cc:492] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:39.761822  2343 tablet_bootstrap.cc:654] T d68da7e811f043dabeb7645bc3d8c4cd P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.764171  2342 tablet_bootstrap.cc:654] T c04e46c4e09740b2a847d140e0a56b12 P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.765626  2337 tablet_bootstrap.cc:654] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.766896  2342 tablet_bootstrap.cc:492] T c04e46c4e09740b2a847d140e0a56b12 P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:39.767031  2342 ts_tablet_manager.cc:1367] T c04e46c4e09740b2a847d140e0a56b12 P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.000s
I20240503 02:11:39.767314  2337 tablet_bootstrap.cc:492] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:39.767467  2337 ts_tablet_manager.cc:1367] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.002s
I20240503 02:11:39.767846  2342 raft_consensus.cc:348] T c04e46c4e09740b2a847d140e0a56b12 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:39.768074  2342 raft_consensus.cc:374] T c04e46c4e09740b2a847d140e0a56b12 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.768186  2342 raft_consensus.cc:729] T c04e46c4e09740b2a847d140e0a56b12 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.768188  2337 raft_consensus.cc:348] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:39.768437  2337 raft_consensus.cc:374] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.768553  2337 raft_consensus.cc:729] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.768857  2337 consensus_queue.cc:260] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:39.769066  2343 tablet_bootstrap.cc:492] T d68da7e811f043dabeb7645bc3d8c4cd P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:39.769220  2343 ts_tablet_manager.cc:1367] T d68da7e811f043dabeb7645bc3d8c4cd P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.029s	user 0.001s	sys 0.003s
I20240503 02:11:39.769330  2337 ts_tablet_manager.cc:1398] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.002s
I20240503 02:11:39.770038  2343 raft_consensus.cc:348] T d68da7e811f043dabeb7645bc3d8c4cd P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:39.770551  2343 raft_consensus.cc:374] T d68da7e811f043dabeb7645bc3d8c4cd P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.770740  2343 raft_consensus.cc:729] T d68da7e811f043dabeb7645bc3d8c4cd P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.771201  2343 consensus_queue.cc:260] T d68da7e811f043dabeb7645bc3d8c4cd P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:39.771772  2343 ts_tablet_manager.cc:1398] T d68da7e811f043dabeb7645bc3d8c4cd P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:39.771924  2342 consensus_queue.cc:260] T c04e46c4e09740b2a847d140e0a56b12 P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:39.772675  2342 ts_tablet_manager.cc:1398] T c04e46c4e09740b2a847d140e0a56b12 P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240503 02:11:39.773111  2342 tablet_bootstrap.cc:492] T 0751c6ca11054c0f93d45574c49dc1ac P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:39.774662  2342 tablet_bootstrap.cc:654] T 0751c6ca11054c0f93d45574c49dc1ac P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.777040  2342 tablet_bootstrap.cc:492] T 0751c6ca11054c0f93d45574c49dc1ac P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:39.777259  2342 ts_tablet_manager.cc:1367] T 0751c6ca11054c0f93d45574c49dc1ac P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 02:11:39.778124  2342 raft_consensus.cc:348] T 0751c6ca11054c0f93d45574c49dc1ac P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:39.778322  2342 raft_consensus.cc:374] T 0751c6ca11054c0f93d45574c49dc1ac P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.778395  2342 raft_consensus.cc:729] T 0751c6ca11054c0f93d45574c49dc1ac P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.778577  2342 consensus_queue.cc:260] T 0751c6ca11054c0f93d45574c49dc1ac P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:39.779110  2342 ts_tablet_manager.cc:1398] T 0751c6ca11054c0f93d45574c49dc1ac P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:39.779510  2219 raft_consensus.cc:482] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:39.779686  2219 raft_consensus.cc:504] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:39.780318  2219 leader_election.cc:288] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773), 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485)
I20240503 02:11:39.780766  1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0751c6ca11054c0f93d45574c49dc1ac" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:39.780973  1157 raft_consensus.cc:2454] T 0751c6ca11054c0f93d45574c49dc1ac P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:39.781366   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0751c6ca11054c0f93d45574c49dc1ac" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" is_pre_election: true
I20240503 02:11:39.781381  1355 leader_election.cc:302] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 4fda3b525c234f8f88aaefd355e7274e; no voters: 
I20240503 02:11:39.781590   895 raft_consensus.cc:2454] T 0751c6ca11054c0f93d45574c49dc1ac P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:39.781716  2219 raft_consensus.cc:2781] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:39.781827  2219 raft_consensus.cc:482] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:39.781921  2219 raft_consensus.cc:3037] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.785296  2219 raft_consensus.cc:504] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:39.785985  2219 leader_election.cc:288] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Requested vote from peers 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773), 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485)
I20240503 02:11:39.787290  1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0751c6ca11054c0f93d45574c49dc1ac" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:39.787519  1157 raft_consensus.cc:3037] T 0751c6ca11054c0f93d45574c49dc1ac P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.787448   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0751c6ca11054c0f93d45574c49dc1ac" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
I20240503 02:11:39.787667   895 raft_consensus.cc:3037] T 0751c6ca11054c0f93d45574c49dc1ac P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.849254  2192 raft_consensus.cc:482] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:39.849465  2192 raft_consensus.cc:504] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:39.850242  2192 leader_election.cc:288] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:39.850754   763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68da7e811f043dabeb7645bc3d8c4cd" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177" is_pre_election: true
I20240503 02:11:39.850996   763 raft_consensus.cc:2454] T d68da7e811f043dabeb7645bc3d8c4cd P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 0.
I20240503 02:11:39.850927  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68da7e811f043dabeb7645bc3d8c4cd" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f" is_pre_election: true
I20240503 02:11:39.851171  1027 raft_consensus.cc:2454] T d68da7e811f043dabeb7645bc3d8c4cd P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 0.
I20240503 02:11:39.851460  1223 leader_election.cc:302] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, e8d9a43983f2478ea7ababa09b230177; no voters: 
I20240503 02:11:39.851754  2192 raft_consensus.cc:2781] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:39.851894  2192 raft_consensus.cc:482] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:39.851990  2192 raft_consensus.cc:3037] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.883832  2219 raft_consensus.cc:482] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:39.884014  2219 raft_consensus.cc:504] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:39.884665  1157 raft_consensus.cc:2454] T 0751c6ca11054c0f93d45574c49dc1ac P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:39.884725  2219 leader_election.cc:288] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:39.884690   895 raft_consensus.cc:2454] T 0751c6ca11054c0f93d45574c49dc1ac P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:39.885207  1355 leader_election.cc:302] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:39.885460   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04e46c4e09740b2a847d140e0a56b12" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" is_pre_election: true
I20240503 02:11:39.885493  1157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04e46c4e09740b2a847d140e0a56b12" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:39.885705   895 raft_consensus.cc:2454] T c04e46c4e09740b2a847d140e0a56b12 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:39.885763  1157 raft_consensus.cc:2454] T c04e46c4e09740b2a847d140e0a56b12 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:39.885877  2192 raft_consensus.cc:504] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:39.886210  1355 leader_election.cc:302] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 4fda3b525c234f8f88aaefd355e7274e; no voters: 
I20240503 02:11:39.886713  2192 leader_election.cc:288] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 election: Requested vote from peers e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:39.886202  2219 raft_consensus.cc:2781] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:39.887758  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68da7e811f043dabeb7645bc3d8c4cd" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f"
I20240503 02:11:39.887995  1027 raft_consensus.cc:3037] T d68da7e811f043dabeb7645bc3d8c4cd P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.890993   763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68da7e811f043dabeb7645bc3d8c4cd" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177"
I20240503 02:11:39.891216   763 raft_consensus.cc:3037] T d68da7e811f043dabeb7645bc3d8c4cd P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.891897  2353 raft_consensus.cc:2781] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:39.892004  2353 raft_consensus.cc:482] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:39.892127  2353 raft_consensus.cc:3037] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.893193  2219 raft_consensus.cc:686] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [term 1 LEADER]: Becoming Leader. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Running, Role: LEADER
I20240503 02:11:39.893496  2219 consensus_queue.cc:237] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } }
I20240503 02:11:39.897027   618 catalog_manager.cc:5484] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6). New cstate: current_term: 1 leader_uuid: "4fda3b525c234f8f88aaefd355e7274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:39.986397   763 raft_consensus.cc:2454] T d68da7e811f043dabeb7645bc3d8c4cd P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 1.
I20240503 02:11:39.986377  1027 raft_consensus.cc:2454] T d68da7e811f043dabeb7645bc3d8c4cd P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 1.
I20240503 02:11:39.987176  1223 leader_election.cc:302] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, e8d9a43983f2478ea7ababa09b230177; no voters: 
I20240503 02:11:39.987931  2192 raft_consensus.cc:2781] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:39.988082  2192 raft_consensus.cc:686] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [term 1 LEADER]: Becoming Leader. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Running, Role: LEADER
I20240503 02:11:39.988291  2353 raft_consensus.cc:504] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:39.988413  2192 consensus_queue.cc:237] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:39.989765  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04e46c4e09740b2a847d140e0a56b12" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:39.990001  1158 raft_consensus.cc:3037] T c04e46c4e09740b2a847d140e0a56b12 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.991030   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c04e46c4e09740b2a847d140e0a56b12" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
I20240503 02:11:39.991271   895 raft_consensus.cc:3037] T c04e46c4e09740b2a847d140e0a56b12 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.994391  2353 leader_election.cc:288] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Requested vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:39.995838   618 catalog_manager.cc:5484] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5). New cstate: current_term: 1 leader_uuid: "572709bd9ae041a4a98029298a6f69c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:40.085860  1158 raft_consensus.cc:2454] T c04e46c4e09740b2a847d140e0a56b12 P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:40.085875   895 raft_consensus.cc:2454] T c04e46c4e09740b2a847d140e0a56b12 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:40.086409  1355 leader_election.cc:302] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:40.086902  2219 raft_consensus.cc:2781] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:40.087097  2219 raft_consensus.cc:686] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [term 1 LEADER]: Becoming Leader. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Running, Role: LEADER
I20240503 02:11:40.087625  2219 consensus_queue.cc:237] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:40.092610   618 catalog_manager.cc:5484] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6). New cstate: current_term: 1 leader_uuid: "4fda3b525c234f8f88aaefd355e7274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:40.364765  2219 consensus_queue.cc:1035] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:40.367939  2352 consensus_queue.cc:1035] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:40.391279  2192 consensus_queue.cc:1035] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:40.393496  2353 consensus_queue.cc:1035] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:40.465698  2353 consensus_queue.cc:1035] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:40.472568  2354 consensus_queue.cc:1035] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:40.587422  2359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:40.587739  2359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:40.597321  2359 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'

INSERT report
    rows total: 2000
    time total: 244.822 ms
  time per row: 0.122411 ms
I20240503 02:11:41.001864   614 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44878:
name: "rebalance_test_table_10"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:41.012709  1268 tablet_service.cc:1467] Processing CreateTablet for tablet ff7268bf9a9b47bca0168ec2e37122e8 (DEFAULT_TABLE table=rebalance_test_table_10 [id=64134d6ea4384dd6b59e891ec655cae7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:41.012851   743 tablet_service.cc:1467] Processing CreateTablet for tablet ff7268bf9a9b47bca0168ec2e37122e8 (DEFAULT_TABLE table=rebalance_test_table_10 [id=64134d6ea4384dd6b59e891ec655cae7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:41.013046  1006 tablet_service.cc:1467] Processing CreateTablet for tablet ff7268bf9a9b47bca0168ec2e37122e8 (DEFAULT_TABLE table=rebalance_test_table_10 [id=64134d6ea4384dd6b59e891ec655cae7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:41.013258  1268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7268bf9a9b47bca0168ec2e37122e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:41.013312   743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7268bf9a9b47bca0168ec2e37122e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:41.013453  1006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff7268bf9a9b47bca0168ec2e37122e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:41.020839  1399 tablet_service.cc:1467] Processing CreateTablet for tablet d0e1479d6cb4465e8c29fb5c58415421 (DEFAULT_TABLE table=rebalance_test_table_10 [id=64134d6ea4384dd6b59e891ec655cae7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:41.021344  1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e1479d6cb4465e8c29fb5c58415421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:41.024046   876 tablet_service.cc:1467] Processing CreateTablet for tablet d0e1479d6cb4465e8c29fb5c58415421 (DEFAULT_TABLE table=rebalance_test_table_10 [id=64134d6ea4384dd6b59e891ec655cae7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:41.024603   876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e1479d6cb4465e8c29fb5c58415421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:41.031144   744 tablet_service.cc:1467] Processing CreateTablet for tablet ab506e9c0d6e42d69ce2cb7f4692e0b7 (DEFAULT_TABLE table=rebalance_test_table_10 [id=64134d6ea4384dd6b59e891ec655cae7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:41.031802   744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab506e9c0d6e42d69ce2cb7f4692e0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:41.035260  1269 tablet_service.cc:1467] Processing CreateTablet for tablet ab506e9c0d6e42d69ce2cb7f4692e0b7 (DEFAULT_TABLE table=rebalance_test_table_10 [id=64134d6ea4384dd6b59e891ec655cae7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:41.035867  1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab506e9c0d6e42d69ce2cb7f4692e0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:41.036921  1007 tablet_service.cc:1467] Processing CreateTablet for tablet ab506e9c0d6e42d69ce2cb7f4692e0b7 (DEFAULT_TABLE table=rebalance_test_table_10 [id=64134d6ea4384dd6b59e891ec655cae7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:41.037606  1007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab506e9c0d6e42d69ce2cb7f4692e0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:41.042825  2428 tablet_bootstrap.cc:492] T ff7268bf9a9b47bca0168ec2e37122e8 P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:41.048207  2428 tablet_bootstrap.cc:654] T ff7268bf9a9b47bca0168ec2e37122e8 P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:41.065053  2431 tablet_bootstrap.cc:492] T ff7268bf9a9b47bca0168ec2e37122e8 P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:41.067083  2431 tablet_bootstrap.cc:654] T ff7268bf9a9b47bca0168ec2e37122e8 P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:41.070569  2428 tablet_bootstrap.cc:492] T ff7268bf9a9b47bca0168ec2e37122e8 P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:41.071547  2428 ts_tablet_manager.cc:1367] T ff7268bf9a9b47bca0168ec2e37122e8 P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.029s	user 0.005s	sys 0.007s
I20240503 02:11:41.072999  2428 raft_consensus.cc:348] T ff7268bf9a9b47bca0168ec2e37122e8 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.075300  2433 tablet_bootstrap.cc:492] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:41.076913  2428 raft_consensus.cc:374] T ff7268bf9a9b47bca0168ec2e37122e8 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:41.077332  2428 raft_consensus.cc:729] T ff7268bf9a9b47bca0168ec2e37122e8 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:41.077633  2433 tablet_bootstrap.cc:654] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:41.077958  2428 consensus_queue.cc:260] T ff7268bf9a9b47bca0168ec2e37122e8 P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.078671  2431 tablet_bootstrap.cc:492] T ff7268bf9a9b47bca0168ec2e37122e8 P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:41.079248  2428 ts_tablet_manager.cc:1398] T ff7268bf9a9b47bca0168ec2e37122e8 P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.002s
I20240503 02:11:41.080024  2428 tablet_bootstrap.cc:492] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:41.081745  2431 ts_tablet_manager.cc:1367] T ff7268bf9a9b47bca0168ec2e37122e8 P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.017s	user 0.002s	sys 0.004s
I20240503 02:11:41.083014  2430 tablet_bootstrap.cc:492] T d0e1479d6cb4465e8c29fb5c58415421 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:41.083581  2431 raft_consensus.cc:348] T ff7268bf9a9b47bca0168ec2e37122e8 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.083796  2431 raft_consensus.cc:374] T ff7268bf9a9b47bca0168ec2e37122e8 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:41.083905  2431 raft_consensus.cc:729] T ff7268bf9a9b47bca0168ec2e37122e8 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:41.084133  2431 consensus_queue.cc:260] T ff7268bf9a9b47bca0168ec2e37122e8 P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.084668  2431 ts_tablet_manager.cc:1398] T ff7268bf9a9b47bca0168ec2e37122e8 P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:11:41.086025  2428 tablet_bootstrap.cc:654] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:41.086267  1137 tablet_service.cc:1467] Processing CreateTablet for tablet d0e1479d6cb4465e8c29fb5c58415421 (DEFAULT_TABLE table=rebalance_test_table_10 [id=64134d6ea4384dd6b59e891ec655cae7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:41.086874  1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e1479d6cb4465e8c29fb5c58415421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:41.088110  2428 tablet_bootstrap.cc:492] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:41.088264  2428 ts_tablet_manager.cc:1367] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.000s
I20240503 02:11:41.088481  2431 tablet_bootstrap.cc:492] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:41.089071  2428 raft_consensus.cc:348] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.089303  2428 raft_consensus.cc:374] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:41.089373  2428 raft_consensus.cc:729] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:41.089664  2428 consensus_queue.cc:260] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.090154  2431 tablet_bootstrap.cc:654] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:41.093555  2428 ts_tablet_manager.cc:1398] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.001s
I20240503 02:11:41.096797  2436 tablet_bootstrap.cc:492] T d0e1479d6cb4465e8c29fb5c58415421 P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:41.097939  2430 tablet_bootstrap.cc:654] T d0e1479d6cb4465e8c29fb5c58415421 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:41.098646  2436 tablet_bootstrap.cc:654] T d0e1479d6cb4465e8c29fb5c58415421 P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:41.107767  2430 tablet_bootstrap.cc:492] T d0e1479d6cb4465e8c29fb5c58415421 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:41.108136  2430 ts_tablet_manager.cc:1367] T d0e1479d6cb4465e8c29fb5c58415421 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.025s	user 0.003s	sys 0.002s
I20240503 02:11:41.116312  2427 tablet_bootstrap.cc:492] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:41.129351  2433 tablet_bootstrap.cc:492] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:41.129688  2433 ts_tablet_manager.cc:1367] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.055s	user 0.005s	sys 0.000s
I20240503 02:11:41.130554  2427 tablet_bootstrap.cc:654] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:41.130932  2433 raft_consensus.cc:348] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:41.131392  2433 raft_consensus.cc:374] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:41.131567  2433 raft_consensus.cc:729] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:41.131502  2430 raft_consensus.cc:348] T d0e1479d6cb4465e8c29fb5c58415421 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:41.132696  2430 raft_consensus.cc:374] T d0e1479d6cb4465e8c29fb5c58415421 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:41.132925  2430 raft_consensus.cc:729] T d0e1479d6cb4465e8c29fb5c58415421 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:41.133064  2433 consensus_queue.cc:260] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:41.133567  2431 tablet_bootstrap.cc:492] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:41.133808  2431 ts_tablet_manager.cc:1367] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.045s	user 0.017s	sys 0.010s
I20240503 02:11:41.134799  2431 raft_consensus.cc:348] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.134971  2433 ts_tablet_manager.cc:1398] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:11:41.135138  2431 raft_consensus.cc:374] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:41.135259  2431 raft_consensus.cc:729] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:41.135546  2431 consensus_queue.cc:260] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.135708  2430 consensus_queue.cc:260] T d0e1479d6cb4465e8c29fb5c58415421 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:41.139135  2431 ts_tablet_manager.cc:1398] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:11:41.141840  2430 ts_tablet_manager.cc:1398] T d0e1479d6cb4465e8c29fb5c58415421 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.033s	user 0.002s	sys 0.002s
I20240503 02:11:41.146862  2427 tablet_bootstrap.cc:492] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:41.147168  2427 ts_tablet_manager.cc:1367] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.031s	user 0.005s	sys 0.000s
I20240503 02:11:41.147275  2436 tablet_bootstrap.cc:492] T d0e1479d6cb4465e8c29fb5c58415421 P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:41.147596  2436 ts_tablet_manager.cc:1367] T d0e1479d6cb4465e8c29fb5c58415421 P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.051s	user 0.012s	sys 0.028s
I20240503 02:11:41.148308  2427 raft_consensus.cc:348] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.148530  2427 raft_consensus.cc:374] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:41.148643  2427 raft_consensus.cc:729] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:41.148567  2436 raft_consensus.cc:348] T d0e1479d6cb4465e8c29fb5c58415421 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:41.148870  2436 raft_consensus.cc:374] T d0e1479d6cb4465e8c29fb5c58415421 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:41.149021  2436 raft_consensus.cc:729] T d0e1479d6cb4465e8c29fb5c58415421 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:41.148919  2427 consensus_queue.cc:260] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.149273  2421 raft_consensus.cc:482] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:41.149448  2421 raft_consensus.cc:504] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:41.149917  2427 ts_tablet_manager.cc:1398] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:41.150249  2427 tablet_bootstrap.cc:492] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:41.150301  2421 leader_election.cc:288] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:41.152129  2427 tablet_bootstrap.cc:654] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:41.152098  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1479d6cb4465e8c29fb5c58415421" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:41.153038  2192 raft_consensus.cc:482] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:41.153219  2192 raft_consensus.cc:504] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.154327  2192 leader_election.cc:288] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:41.155627  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab506e9c0d6e42d69ce2cb7f4692e0b7" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f" is_pre_election: true
I20240503 02:11:41.155965  1027 raft_consensus.cc:2454] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 0.
I20240503 02:11:41.156697  1223 leader_election.cc:302] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:41.158011  2352 raft_consensus.cc:2781] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:41.158136  2352 raft_consensus.cc:482] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:41.158198  2352 raft_consensus.cc:3037] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:41.158803   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1479d6cb4465e8c29fb5c58415421" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" is_pre_election: true
I20240503 02:11:41.159086   895 raft_consensus.cc:2454] T d0e1479d6cb4465e8c29fb5c58415421 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:41.149358  2436 consensus_queue.cc:260] T d0e1479d6cb4465e8c29fb5c58415421 P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:41.159184   764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab506e9c0d6e42d69ce2cb7f4692e0b7" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177" is_pre_election: true
I20240503 02:11:41.159677  1156 raft_consensus.cc:2454] T d0e1479d6cb4465e8c29fb5c58415421 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:41.159819  2427 tablet_bootstrap.cc:492] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:41.160005  2427 ts_tablet_manager.cc:1367] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.001s
I20240503 02:11:41.160162  2436 ts_tablet_manager.cc:1398] T d0e1479d6cb4465e8c29fb5c58415421 P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.012s	user 0.000s	sys 0.002s
I20240503 02:11:41.160754  2427 raft_consensus.cc:348] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.160972  2427 raft_consensus.cc:374] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:41.161065  2427 raft_consensus.cc:729] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:41.161274  1355 leader_election.cc:302] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 4fda3b525c234f8f88aaefd355e7274e; no voters: 
I20240503 02:11:41.161348  2427 consensus_queue.cc:260] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
W20240503 02:11:41.161528  1223 leader_election.cc:341] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:41.162025  2421 raft_consensus.cc:2781] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:41.162039  2427 ts_tablet_manager.cc:1398] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:11:41.162142  2421 raft_consensus.cc:482] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:41.162233  2421 raft_consensus.cc:3037] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:41.222267  2421 raft_consensus.cc:504] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:41.222288  2352 raft_consensus.cc:504] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.223027  2421 leader_election.cc:288] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Requested vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:41.223574   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1479d6cb4465e8c29fb5c58415421" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
I20240503 02:11:41.223801   895 raft_consensus.cc:3037] T d0e1479d6cb4465e8c29fb5c58415421 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:41.223719  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e1479d6cb4465e8c29fb5c58415421" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:41.223951  1156 raft_consensus.cc:3037] T d0e1479d6cb4465e8c29fb5c58415421 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:41.223884   764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab506e9c0d6e42d69ce2cb7f4692e0b7" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177"
I20240503 02:11:41.224108   764 raft_consensus.cc:3037] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:41.226917   764 raft_consensus.cc:2454] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 1.
I20240503 02:11:41.226926  1156 raft_consensus.cc:2454] T d0e1479d6cb4465e8c29fb5c58415421 P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:41.227187   895 raft_consensus.cc:2454] T d0e1479d6cb4465e8c29fb5c58415421 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:41.227469  1223 leader_election.cc:302] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, e8d9a43983f2478ea7ababa09b230177; no voters: 
I20240503 02:11:41.227463  1355 leader_election.cc:302] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 4fda3b525c234f8f88aaefd355e7274e; no voters: 
I20240503 02:11:41.227835  2352 leader_election.cc:288] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 election: Requested vote from peers e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:41.227985  2421 raft_consensus.cc:2781] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:41.227847  2192 raft_consensus.cc:2781] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:41.228298  2192 raft_consensus.cc:686] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [term 1 LEADER]: Becoming Leader. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Running, Role: LEADER
I20240503 02:11:41.228484  2421 raft_consensus.cc:686] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [term 1 LEADER]: Becoming Leader. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Running, Role: LEADER
I20240503 02:11:41.228598  2192 consensus_queue.cc:237] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.228737  2421 consensus_queue.cc:237] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } }
I20240503 02:11:41.229861  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab506e9c0d6e42d69ce2cb7f4692e0b7" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f"
I20240503 02:11:41.230147  1027 raft_consensus.cc:3037] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:41.232547   621 catalog_manager.cc:5484] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6). New cstate: current_term: 1 leader_uuid: "4fda3b525c234f8f88aaefd355e7274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:41.233769   617 catalog_manager.cc:5484] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5). New cstate: current_term: 1 leader_uuid: "572709bd9ae041a4a98029298a6f69c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:41.257602  2209 raft_consensus.cc:482] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:41.257851  2209 raft_consensus.cc:504] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.258991  2209 leader_election.cc:288] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:41.259474  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7268bf9a9b47bca0168ec2e37122e8" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1" is_pre_election: true
I20240503 02:11:41.259581  1026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7268bf9a9b47bca0168ec2e37122e8" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f" is_pre_election: true
I20240503 02:11:41.259719  1288 raft_consensus.cc:2454] T ff7268bf9a9b47bca0168ec2e37122e8 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 0.
I20240503 02:11:41.259855  1026 raft_consensus.cc:2454] T ff7268bf9a9b47bca0168ec2e37122e8 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 0.
I20240503 02:11:41.260282   699 leader_election.cc:302] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, e8d9a43983f2478ea7ababa09b230177; no voters: 
I20240503 02:11:41.260608  2209 raft_consensus.cc:2781] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:41.260735  2209 raft_consensus.cc:482] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:41.260828  2209 raft_consensus.cc:3037] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:41.323702  1027 raft_consensus.cc:2454] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 1.
I20240503 02:11:41.323761  2209 raft_consensus.cc:504] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.324676  2209 leader_election.cc:288] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 election: Requested vote from peers 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5:41189), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:41.325119  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7268bf9a9b47bca0168ec2e37122e8" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f"
I20240503 02:11:41.325383  1027 raft_consensus.cc:3037] T ff7268bf9a9b47bca0168ec2e37122e8 P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:41.329088  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff7268bf9a9b47bca0168ec2e37122e8" candidate_uuid: "e8d9a43983f2478ea7ababa09b230177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572709bd9ae041a4a98029298a6f69c1"
I20240503 02:11:41.329394  1288 raft_consensus.cc:3037] T ff7268bf9a9b47bca0168ec2e37122e8 P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:41.429634  1288 raft_consensus.cc:2454] T ff7268bf9a9b47bca0168ec2e37122e8 P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 1.
I20240503 02:11:41.429656  1027 raft_consensus.cc:2454] T ff7268bf9a9b47bca0168ec2e37122e8 P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d9a43983f2478ea7ababa09b230177 in term 1.
I20240503 02:11:41.430270   699 leader_election.cc:302] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, e8d9a43983f2478ea7ababa09b230177; no voters: 
I20240503 02:11:41.430658  2209 raft_consensus.cc:2781] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:41.430824  2209 raft_consensus.cc:686] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [term 1 LEADER]: Becoming Leader. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Running, Role: LEADER
I20240503 02:11:41.431154  2209 consensus_queue.cc:237] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } }
I20240503 02:11:41.434585   617 catalog_manager.cc:5484] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8d9a43983f2478ea7ababa09b230177 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "e8d9a43983f2478ea7ababa09b230177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:41.642716  2192 consensus_queue.cc:1035] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:41.649650  2352 consensus_queue.cc:1035] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:41.674084  2354 consensus_queue.cc:1035] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:41.681826  2354 consensus_queue.cc:1035] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:41.885272  2447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:41.885689  2447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:41.895388  2447 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:11:41.895588  2445 consensus_queue.cc:1035] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:41.901895  2445 consensus_queue.cc:1035] T ff7268bf9a9b47bca0168ec2e37122e8 P e8d9a43983f2478ea7ababa09b230177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_10'
W20240503 02:11:42.223475   961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.1:34401, user_credentials={real_user=slave}} blocked reactor thread for 74093.5us

INSERT report
    rows total: 2000
    time total: 379.303 ms
  time per row: 0.189652 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240503 02:11:43.444518   617 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44878:
name: "rebalance_test_table_11"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:43.455174   876 tablet_service.cc:1467] Processing CreateTablet for tablet 824bfd1d47374f10b3b63023d9b5e051 (DEFAULT_TABLE table=rebalance_test_table_11 [id=ca7bd24922cf4df59fd8b715c0a421cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:43.455417  1137 tablet_service.cc:1467] Processing CreateTablet for tablet 824bfd1d47374f10b3b63023d9b5e051 (DEFAULT_TABLE table=rebalance_test_table_11 [id=ca7bd24922cf4df59fd8b715c0a421cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:43.455720   876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824bfd1d47374f10b3b63023d9b5e051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.455868  1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824bfd1d47374f10b3b63023d9b5e051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.455997  1399 tablet_service.cc:1467] Processing CreateTablet for tablet 824bfd1d47374f10b3b63023d9b5e051 (DEFAULT_TABLE table=rebalance_test_table_11 [id=ca7bd24922cf4df59fd8b715c0a421cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:43.456419  1399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824bfd1d47374f10b3b63023d9b5e051. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.459339  1138 tablet_service.cc:1467] Processing CreateTablet for tablet c1d5289a99ca41dfba0ddaf9fbf6a72c (DEFAULT_TABLE table=rebalance_test_table_11 [id=ca7bd24922cf4df59fd8b715c0a421cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:43.459510   743 tablet_service.cc:1467] Processing CreateTablet for tablet aaeb0eedcd0345a3aadb3f2fc9c484ba (DEFAULT_TABLE table=rebalance_test_table_11 [id=ca7bd24922cf4df59fd8b715c0a421cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:43.459527  1006 tablet_service.cc:1467] Processing CreateTablet for tablet aaeb0eedcd0345a3aadb3f2fc9c484ba (DEFAULT_TABLE table=rebalance_test_table_11 [id=ca7bd24922cf4df59fd8b715c0a421cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:43.459744  1138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d5289a99ca41dfba0ddaf9fbf6a72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.459923  1006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaeb0eedcd0345a3aadb3f2fc9c484ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.459929   743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaeb0eedcd0345a3aadb3f2fc9c484ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.463464   875 tablet_service.cc:1467] Processing CreateTablet for tablet c1d5289a99ca41dfba0ddaf9fbf6a72c (DEFAULT_TABLE table=rebalance_test_table_11 [id=ca7bd24922cf4df59fd8b715c0a421cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:43.463959   875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d5289a99ca41dfba0ddaf9fbf6a72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.464530  1400 tablet_service.cc:1467] Processing CreateTablet for tablet c1d5289a99ca41dfba0ddaf9fbf6a72c (DEFAULT_TABLE table=rebalance_test_table_11 [id=ca7bd24922cf4df59fd8b715c0a421cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:43.464650  1269 tablet_service.cc:1467] Processing CreateTablet for tablet aaeb0eedcd0345a3aadb3f2fc9c484ba (DEFAULT_TABLE table=rebalance_test_table_11 [id=ca7bd24922cf4df59fd8b715c0a421cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:43.464912  1400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1d5289a99ca41dfba0ddaf9fbf6a72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.465073  1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaeb0eedcd0345a3aadb3f2fc9c484ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:43.478394  2528 tablet_bootstrap.cc:492] T 824bfd1d47374f10b3b63023d9b5e051 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:43.479809  2525 tablet_bootstrap.cc:492] T aaeb0eedcd0345a3aadb3f2fc9c484ba P f5e38dcb24084a1e969f83bf34aef88f: Bootstrap starting.
I20240503 02:11:43.482060  2525 tablet_bootstrap.cc:654] T aaeb0eedcd0345a3aadb3f2fc9c484ba P f5e38dcb24084a1e969f83bf34aef88f: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.483170  2528 tablet_bootstrap.cc:654] T 824bfd1d47374f10b3b63023d9b5e051 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.487165  2527 tablet_bootstrap.cc:492] T aaeb0eedcd0345a3aadb3f2fc9c484ba P e8d9a43983f2478ea7ababa09b230177: Bootstrap starting.
I20240503 02:11:43.487344  2529 tablet_bootstrap.cc:492] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1: Bootstrap starting.
I20240503 02:11:43.488155  2526 tablet_bootstrap.cc:492] T 824bfd1d47374f10b3b63023d9b5e051 P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:43.489310  2529 tablet_bootstrap.cc:654] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.489466  2527 tablet_bootstrap.cc:654] T aaeb0eedcd0345a3aadb3f2fc9c484ba P e8d9a43983f2478ea7ababa09b230177: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.489967  2526 tablet_bootstrap.cc:654] T 824bfd1d47374f10b3b63023d9b5e051 P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.490041  2524 tablet_bootstrap.cc:492] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:43.491451  2525 tablet_bootstrap.cc:492] T aaeb0eedcd0345a3aadb3f2fc9c484ba P f5e38dcb24084a1e969f83bf34aef88f: No bootstrap required, opened a new log
I20240503 02:11:43.491672  2525 ts_tablet_manager.cc:1367] T aaeb0eedcd0345a3aadb3f2fc9c484ba P f5e38dcb24084a1e969f83bf34aef88f: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.000s
I20240503 02:11:43.491809  2524 tablet_bootstrap.cc:654] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.492642  2525 raft_consensus.cc:348] T aaeb0eedcd0345a3aadb3f2fc9c484ba P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:43.492936  2525 raft_consensus.cc:374] T aaeb0eedcd0345a3aadb3f2fc9c484ba P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.493077  2525 raft_consensus.cc:729] T aaeb0eedcd0345a3aadb3f2fc9c484ba P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e38dcb24084a1e969f83bf34aef88f, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.493373  2528 tablet_bootstrap.cc:492] T 824bfd1d47374f10b3b63023d9b5e051 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:43.493422  2525 consensus_queue.cc:260] T aaeb0eedcd0345a3aadb3f2fc9c484ba P f5e38dcb24084a1e969f83bf34aef88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:43.493706  2528 ts_tablet_manager.cc:1367] T 824bfd1d47374f10b3b63023d9b5e051 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.000s
I20240503 02:11:43.494362  2525 ts_tablet_manager.cc:1398] T aaeb0eedcd0345a3aadb3f2fc9c484ba P f5e38dcb24084a1e969f83bf34aef88f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:43.494585  2528 raft_consensus.cc:348] T 824bfd1d47374f10b3b63023d9b5e051 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.494877  2528 raft_consensus.cc:374] T 824bfd1d47374f10b3b63023d9b5e051 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.495028  2528 raft_consensus.cc:729] T 824bfd1d47374f10b3b63023d9b5e051 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.495347  2528 consensus_queue.cc:260] T 824bfd1d47374f10b3b63023d9b5e051 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.495715  2524 tablet_bootstrap.cc:492] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:43.495894  2524 ts_tablet_manager.cc:1367] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.002s
I20240503 02:11:43.495918  2528 ts_tablet_manager.cc:1398] T 824bfd1d47374f10b3b63023d9b5e051 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 02:11:43.496364  2528 tablet_bootstrap.cc:492] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Bootstrap starting.
I20240503 02:11:43.496737  2524 raft_consensus.cc:348] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.496909  2524 raft_consensus.cc:374] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.497045  2524 raft_consensus.cc:729] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.497267  2524 consensus_queue.cc:260] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.498031  2524 ts_tablet_manager.cc:1398] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:11:43.498309  2524 tablet_bootstrap.cc:492] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e: Bootstrap starting.
I20240503 02:11:43.502593  2528 tablet_bootstrap.cc:654] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.503562  2524 tablet_bootstrap.cc:654] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.504645  2529 tablet_bootstrap.cc:492] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1: No bootstrap required, opened a new log
I20240503 02:11:43.504873  2529 ts_tablet_manager.cc:1367] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1: Time spent bootstrapping tablet: real 0.018s	user 0.004s	sys 0.000s
I20240503 02:11:43.505757  2529 raft_consensus.cc:348] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:43.506080  2529 raft_consensus.cc:374] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.506229  2529 raft_consensus.cc:729] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.506583  2529 consensus_queue.cc:260] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:43.507325  2529 ts_tablet_manager.cc:1398] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:11:43.508945  2527 tablet_bootstrap.cc:492] T aaeb0eedcd0345a3aadb3f2fc9c484ba P e8d9a43983f2478ea7ababa09b230177: No bootstrap required, opened a new log
I20240503 02:11:43.509183  2527 ts_tablet_manager.cc:1367] T aaeb0eedcd0345a3aadb3f2fc9c484ba P e8d9a43983f2478ea7ababa09b230177: Time spent bootstrapping tablet: real 0.022s	user 0.003s	sys 0.006s
I20240503 02:11:43.510489  2528 tablet_bootstrap.cc:492] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: No bootstrap required, opened a new log
I20240503 02:11:43.510413  2527 raft_consensus.cc:348] T aaeb0eedcd0345a3aadb3f2fc9c484ba P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:43.510589  2527 raft_consensus.cc:374] T aaeb0eedcd0345a3aadb3f2fc9c484ba P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.510705  2528 ts_tablet_manager.cc:1367] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent bootstrapping tablet: real 0.014s	user 0.003s	sys 0.002s
I20240503 02:11:43.510749  2527 raft_consensus.cc:729] T aaeb0eedcd0345a3aadb3f2fc9c484ba P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d9a43983f2478ea7ababa09b230177, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.511021  2527 consensus_queue.cc:260] T aaeb0eedcd0345a3aadb3f2fc9c484ba P e8d9a43983f2478ea7ababa09b230177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:43.511616  2527 ts_tablet_manager.cc:1398] T aaeb0eedcd0345a3aadb3f2fc9c484ba P e8d9a43983f2478ea7ababa09b230177: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:11:43.511574  2528 raft_consensus.cc:348] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.511786  2528 raft_consensus.cc:374] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.511860  2528 raft_consensus.cc:729] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88791c3c9c9341a6a2d8a6b5d3aa1f1b, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.512131  2528 consensus_queue.cc:260] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.513795  2528 ts_tablet_manager.cc:1398] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:43.516376  2524 tablet_bootstrap.cc:492] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e: No bootstrap required, opened a new log
I20240503 02:11:43.516571  2524 ts_tablet_manager.cc:1367] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e: Time spent bootstrapping tablet: real 0.018s	user 0.004s	sys 0.001s
I20240503 02:11:43.517205  2524 raft_consensus.cc:348] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.517395  2524 raft_consensus.cc:374] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.517478  2524 raft_consensus.cc:729] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.517789  2524 consensus_queue.cc:260] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.518391  2524 ts_tablet_manager.cc:1398] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:11:43.519989  2526 tablet_bootstrap.cc:492] T 824bfd1d47374f10b3b63023d9b5e051 P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:43.520180  2526 ts_tablet_manager.cc:1367] T 824bfd1d47374f10b3b63023d9b5e051 P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.032s	user 0.000s	sys 0.006s
I20240503 02:11:43.521304  2526 raft_consensus.cc:348] T 824bfd1d47374f10b3b63023d9b5e051 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.521481  2526 raft_consensus.cc:374] T 824bfd1d47374f10b3b63023d9b5e051 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.521581  2526 raft_consensus.cc:729] T 824bfd1d47374f10b3b63023d9b5e051 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.521826  2526 consensus_queue.cc:260] T 824bfd1d47374f10b3b63023d9b5e051 P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.522346  2526 ts_tablet_manager.cc:1398] T 824bfd1d47374f10b3b63023d9b5e051 P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:11:43.522786  2526 tablet_bootstrap.cc:492] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 288540692e7b404991570d9ca16ab6c0: Bootstrap starting.
I20240503 02:11:43.524168  2526 tablet_bootstrap.cc:654] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 288540692e7b404991570d9ca16ab6c0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:43.525992  2526 tablet_bootstrap.cc:492] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 288540692e7b404991570d9ca16ab6c0: No bootstrap required, opened a new log
I20240503 02:11:43.526147  2526 ts_tablet_manager.cc:1367] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 288540692e7b404991570d9ca16ab6c0: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:11:43.526763  2526 raft_consensus.cc:348] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.526945  2526 raft_consensus.cc:374] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:43.527055  2526 raft_consensus.cc:729] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288540692e7b404991570d9ca16ab6c0, State: Initialized, Role: FOLLOWER
I20240503 02:11:43.527309  2526 consensus_queue.cc:260] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 288540692e7b404991570d9ca16ab6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.527732  2526 ts_tablet_manager.cc:1398] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 288540692e7b404991570d9ca16ab6c0: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240503 02:11:43.537606  2507 raft_consensus.cc:482] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:43.537798  2507 raft_consensus.cc:504] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.538448  2507 leader_election.cc:288] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773), 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485)
I20240503 02:11:43.538973  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d5289a99ca41dfba0ddaf9fbf6a72c" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:43.538997   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d5289a99ca41dfba0ddaf9fbf6a72c" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" is_pre_election: true
I20240503 02:11:43.539219  1156 raft_consensus.cc:2454] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:43.539244   895 raft_consensus.cc:2454] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:43.539781  1355 leader_election.cc:302] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:43.540318  2507 raft_consensus.cc:2781] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:43.540463  2507 raft_consensus.cc:482] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:43.540549  2507 raft_consensus.cc:3037] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.556075  2519 raft_consensus.cc:482] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:43.556277  2519 raft_consensus.cc:504] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:43.557140  2519 leader_election.cc:288] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:43.557626   764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeb0eedcd0345a3aadb3f2fc9c484ba" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177" is_pre_election: true
I20240503 02:11:43.557739  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeb0eedcd0345a3aadb3f2fc9c484ba" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f" is_pre_election: true
I20240503 02:11:43.557871   764 raft_consensus.cc:2454] T aaeb0eedcd0345a3aadb3f2fc9c484ba P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 0.
I20240503 02:11:43.558059  1027 raft_consensus.cc:2454] T aaeb0eedcd0345a3aadb3f2fc9c484ba P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 0.
I20240503 02:11:43.558676  1223 leader_election.cc:302] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:43.559036  2519 raft_consensus.cc:2781] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:43.559159  2519 raft_consensus.cc:482] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:43.559231  2519 raft_consensus.cc:3037] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.608234  2507 raft_consensus.cc:504] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.608220  2519 raft_consensus.cc:504] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:43.609427  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d5289a99ca41dfba0ddaf9fbf6a72c" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:43.609606   764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeb0eedcd0345a3aadb3f2fc9c484ba" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8d9a43983f2478ea7ababa09b230177"
I20240503 02:11:43.609690  1156 raft_consensus.cc:3037] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.609823   764 raft_consensus.cc:3037] T aaeb0eedcd0345a3aadb3f2fc9c484ba P e8d9a43983f2478ea7ababa09b230177 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.609750   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1d5289a99ca41dfba0ddaf9fbf6a72c" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
I20240503 02:11:43.610019   895 raft_consensus.cc:3037] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.612314  2507 leader_election.cc:288] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Requested vote from peers 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773), 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485)
I20240503 02:11:43.612524  2507 raft_consensus.cc:482] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:43.612651  2507 raft_consensus.cc:504] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.613008  2519 leader_election.cc:288] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 election: Requested vote from peers e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401), f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511)
I20240503 02:11:43.613621  1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeb0eedcd0345a3aadb3f2fc9c484ba" candidate_uuid: "572709bd9ae041a4a98029298a6f69c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e38dcb24084a1e969f83bf34aef88f"
I20240503 02:11:43.613849  2507 leader_election.cc:288] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:43.613904  1027 raft_consensus.cc:3037] T aaeb0eedcd0345a3aadb3f2fc9c484ba P f5e38dcb24084a1e969f83bf34aef88f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.613844   894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824bfd1d47374f10b3b63023d9b5e051" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" is_pre_election: true
I20240503 02:11:43.614091   894 raft_consensus.cc:2454] T 824bfd1d47374f10b3b63023d9b5e051 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:43.614509  1158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824bfd1d47374f10b3b63023d9b5e051" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0" is_pre_election: true
I20240503 02:11:43.614648  1355 leader_election.cc:302] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fda3b525c234f8f88aaefd355e7274e, 88791c3c9c9341a6a2d8a6b5d3aa1f1b; no voters: 
I20240503 02:11:43.614751  1158 raft_consensus.cc:2454] T 824bfd1d47374f10b3b63023d9b5e051 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 0.
I20240503 02:11:43.614920  2507 raft_consensus.cc:2781] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:43.615178  2507 raft_consensus.cc:482] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:43.615301  2507 raft_consensus.cc:3037] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.706869  1156 raft_consensus.cc:2454] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:43.706908   764 raft_consensus.cc:2454] T aaeb0eedcd0345a3aadb3f2fc9c484ba P e8d9a43983f2478ea7ababa09b230177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 1.
I20240503 02:11:43.706914  1027 raft_consensus.cc:2454] T aaeb0eedcd0345a3aadb3f2fc9c484ba P f5e38dcb24084a1e969f83bf34aef88f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 572709bd9ae041a4a98029298a6f69c1 in term 1.
I20240503 02:11:43.707347   895 raft_consensus.cc:2454] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:43.707449  1223 leader_election.cc:302] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572709bd9ae041a4a98029298a6f69c1, f5e38dcb24084a1e969f83bf34aef88f; no voters: 
I20240503 02:11:43.707585  2507 raft_consensus.cc:504] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.708053  2519 raft_consensus.cc:2781] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:43.708191  2519 raft_consensus.cc:686] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [term 1 LEADER]: Becoming Leader. State: Replica: 572709bd9ae041a4a98029298a6f69c1, State: Running, Role: LEADER
I20240503 02:11:43.708479  1355 leader_election.cc:302] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 4fda3b525c234f8f88aaefd355e7274e; no voters: 
I20240503 02:11:43.708459  2519 consensus_queue.cc:237] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } }
I20240503 02:11:43.708667  2507 leader_election.cc:288] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Requested vote from peers 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485), 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773)
I20240503 02:11:43.708851  2507 raft_consensus.cc:2781] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:43.709429   895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824bfd1d47374f10b3b63023d9b5e051" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b"
I20240503 02:11:43.709676   895 raft_consensus.cc:3037] T 824bfd1d47374f10b3b63023d9b5e051 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.709905  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824bfd1d47374f10b3b63023d9b5e051" candidate_uuid: "4fda3b525c234f8f88aaefd355e7274e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288540692e7b404991570d9ca16ab6c0"
I20240503 02:11:43.710168  1156 raft_consensus.cc:3037] T 824bfd1d47374f10b3b63023d9b5e051 P 288540692e7b404991570d9ca16ab6c0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:43.711419  2507 raft_consensus.cc:686] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [term 1 LEADER]: Becoming Leader. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Running, Role: LEADER
I20240503 02:11:43.711742  2507 consensus_queue.cc:237] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.713814   617 catalog_manager.cc:5484] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 572709bd9ae041a4a98029298a6f69c1 (127.0.145.5). New cstate: current_term: 1 leader_uuid: "572709bd9ae041a4a98029298a6f69c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "572709bd9ae041a4a98029298a6f69c1" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41189 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:43.715878   619 catalog_manager.cc:5484] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6). New cstate: current_term: 1 leader_uuid: "4fda3b525c234f8f88aaefd355e7274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:43.803623  1156 raft_consensus.cc:2454] T 824bfd1d47374f10b3b63023d9b5e051 P 288540692e7b404991570d9ca16ab6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:43.803594   895 raft_consensus.cc:2454] T 824bfd1d47374f10b3b63023d9b5e051 P 88791c3c9c9341a6a2d8a6b5d3aa1f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fda3b525c234f8f88aaefd355e7274e in term 1.
I20240503 02:11:43.804170  1355 leader_election.cc:302] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 288540692e7b404991570d9ca16ab6c0, 4fda3b525c234f8f88aaefd355e7274e; no voters: 
I20240503 02:11:43.804520  2507 raft_consensus.cc:2781] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:43.804769  2507 raft_consensus.cc:686] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [term 1 LEADER]: Becoming Leader. State: Replica: 4fda3b525c234f8f88aaefd355e7274e, State: Running, Role: LEADER
I20240503 02:11:43.805437  2507 consensus_queue.cc:237] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } }
I20240503 02:11:43.808982   619 catalog_manager.cc:5484] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fda3b525c234f8f88aaefd355e7274e (127.0.145.6). New cstate: current_term: 1 leader_uuid: "4fda3b525c234f8f88aaefd355e7274e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fda3b525c234f8f88aaefd355e7274e" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40135 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:44.092397  2519 consensus_queue.cc:1035] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e38dcb24084a1e969f83bf34aef88f" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:44.110924  2507 consensus_queue.cc:1035] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:44.145220  2539 consensus_queue.cc:1035] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:44.147423  2519 consensus_queue.cc:1035] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8d9a43983f2478ea7ababa09b230177" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:44.227903  2543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:44.228263  2543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:44.236589  2543 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:11:44.239889  2550 consensus_queue.cc:1035] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "288540692e7b404991570d9ca16ab6c0" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:44.244428  2539 consensus_queue.cc:1035] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88791c3c9c9341a6a2d8a6b5d3aa1f1b" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_11'
W20240503 02:11:44.487862  1223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.1:34401, user_credentials={real_user=slave}} blocked reactor thread for 67444.3us

INSERT report
    rows total: 2000
    time total: 318.251 ms
  time per row: 0.159125 ms
I20240503 02:11:45.285662  1137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:45.285915  1399 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:11:45.286722  1269 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:45.288115   876 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:45.299605   743 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:45.299669  1006 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:45.594419  1137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:45.605437  1399 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:11:45.615820  1269 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:45.620548   743 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:45.623804  1006 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:45.649078   876 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:45.906808   743 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:45.913220  1137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:45.914520  1399 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:11:45.918118  1006 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:45.921290  1269 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:45.940461   876 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:46.199113  1269 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:46.200791  1137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:46.207399   743 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:46.210258  1006 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:46.211337   876 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:46.230914  1399 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:11:46.528982  1269 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:46.530089  1137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:46.530452  1399 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:11:46.550571   876 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:46.555640   743 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:46.575665  1006 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:46.955626  1399 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:11:46.957422  1137 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:46.961509  1269 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:46.967995   743 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:46.981739   876 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:11:47.002635  1006 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 76b23b9ca28642a885167cbc65e2dc83 | 127.0.145.62:38925 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                    Value                                                                                    |         Tags          |         Master
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                         | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                         | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                        | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                           | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                        | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                        | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                        | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                          | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                         | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.0.145.20:39363 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address      | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
 288540692e7b404991570d9ca16ab6c0 | 127.0.145.4:32773 | HEALTHY | /B       |       2        |       0
 4fda3b525c234f8f88aaefd355e7274e | 127.0.145.6:40135 | HEALTHY | /C       |      10        |       0
 572709bd9ae041a4a98029298a6f69c1 | 127.0.145.5:41189 | HEALTHY | /C       |       7        |       0
 88791c3c9c9341a6a2d8a6b5d3aa1f1b | 127.0.145.2:43485 | HEALTHY | /A       |       5        |       0
 e8d9a43983f2478ea7ababa09b230177 | 127.0.145.1:34401 | HEALTHY | /A       |       7        |       0
 f5e38dcb24084a1e969f83bf34aef88f | 127.0.145.3:35511 | HEALTHY | /B       |       5        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 /A       |       2
 /B       |       2
 /C       |       2

Unusual flags for Tablet Server:
                   Flag                   |                                                                                  Value                                                                                  |         Tags          |      Tablet Server
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                     | experimental          | all 6 server(s) checked
 local_ip_for_outbound_sockets            | 127.0.145.1                                                                                                                                                             | experimental          | 127.0.145.1:34401
 local_ip_for_outbound_sockets            | 127.0.145.2                                                                                                                                                             | experimental          | 127.0.145.2:43485
 local_ip_for_outbound_sockets            | 127.0.145.3                                                                                                                                                             | experimental          | 127.0.145.3:35511
 local_ip_for_outbound_sockets            | 127.0.145.4                                                                                                                                                             | experimental          | 127.0.145.4:32773
 local_ip_for_outbound_sockets            | 127.0.145.5                                                                                                                                                             | experimental          | 127.0.145.5:41189
 local_ip_for_outbound_sockets            | 127.0.145.6                                                                                                                                                             | experimental          | 127.0.145.6:40135
 never_fsync                              | true                                                                                                                                                                    | unsafe,advanced       | all 6 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                       | unsafe,hidden         | all 6 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                    | experimental,advanced | all 6 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                    | unsafe                | all 6 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                      | hidden                | all 6 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.0.145.1:34401
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.0.145.2:43485
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.0.145.3:35511
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.0.145.4:32773
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.0.145.5:41189
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.0.145.6:40135

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.0.145.20:39363 | all 6 server(s) checked
 time_source         | builtin            | all 6 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 7 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name           | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
 rebalance_test_table_0  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_10 | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_11 | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_5  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_6  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_7  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_8  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_9  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 18
 First Quartile | 18
 Median         | 18
 Third Quartile | 18
 Maximum        | 18

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 6
 Tables         | 12
 Tablets        | 36
 Replicas       | 108

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240503 02:11:47.092453   580 log_verifier.cc:126] Checking tablet 0751c6ca11054c0f93d45574c49dc1ac
I20240503 02:11:47.136049   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0751c6ca11054c0f93d45574c49dc1ac
I20240503 02:11:47.136262   580 log_verifier.cc:126] Checking tablet 131e73c6862d4f249f8c9f9bafdcb979
I20240503 02:11:47.173926   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 131e73c6862d4f249f8c9f9bafdcb979
I20240503 02:11:47.174041   580 log_verifier.cc:126] Checking tablet 16a92cef3eb143a5abcceffbd23cad67
I20240503 02:11:47.212821   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 16a92cef3eb143a5abcceffbd23cad67
I20240503 02:11:47.212941   580 log_verifier.cc:126] Checking tablet 16edf1727a1e4c6f810d22bb97872613
I20240503 02:11:47.252409   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 16edf1727a1e4c6f810d22bb97872613
I20240503 02:11:47.252517   580 log_verifier.cc:126] Checking tablet 18ccfc8dbf884ab3a3e3338d679787af
I20240503 02:11:47.294387   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 18ccfc8dbf884ab3a3e3338d679787af
I20240503 02:11:47.294517   580 log_verifier.cc:126] Checking tablet 2686a2ecc7594849ba4f0213764f27a8
I20240503 02:11:47.335955   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2686a2ecc7594849ba4f0213764f27a8
I20240503 02:11:47.336062   580 log_verifier.cc:126] Checking tablet 3c9396ad859e4ea08a06ef08570ac3fb
I20240503 02:11:47.378557   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3c9396ad859e4ea08a06ef08570ac3fb
I20240503 02:11:47.378712   580 log_verifier.cc:126] Checking tablet 3ca4c55ff01a428daaa3866840e9600a
I20240503 02:11:47.424634   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3ca4c55ff01a428daaa3866840e9600a
I20240503 02:11:47.424790   580 log_verifier.cc:126] Checking tablet 3e64997bb292442eb4982967594bd7f7
I20240503 02:11:47.469069   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3e64997bb292442eb4982967594bd7f7
I20240503 02:11:47.469179   580 log_verifier.cc:126] Checking tablet 41a6042ee7784967913fbf65d25ddc28
I20240503 02:11:47.511076   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 41a6042ee7784967913fbf65d25ddc28
I20240503 02:11:47.511198   580 log_verifier.cc:126] Checking tablet 4775eb444281418e830ed6944f5f8ec3
I20240503 02:11:47.553179   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4775eb444281418e830ed6944f5f8ec3
I20240503 02:11:47.553321   580 log_verifier.cc:126] Checking tablet 477c00ddd6e543e7b9da2f9a0561be51
I20240503 02:11:47.602795   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 477c00ddd6e543e7b9da2f9a0561be51
I20240503 02:11:47.602924   580 log_verifier.cc:126] Checking tablet 6947883c90ca46709a67ca185471178e
I20240503 02:11:47.645094   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6947883c90ca46709a67ca185471178e
I20240503 02:11:47.645248   580 log_verifier.cc:126] Checking tablet 6f6cdecb3c1d45dc8775f47bdd265d3c
I20240503 02:11:47.687228   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6f6cdecb3c1d45dc8775f47bdd265d3c
I20240503 02:11:47.687345   580 log_verifier.cc:126] Checking tablet 7e7f9eb8403e45d88a5532506d06e2e2
I20240503 02:11:47.727891   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7e7f9eb8403e45d88a5532506d06e2e2
I20240503 02:11:47.728039   580 log_verifier.cc:126] Checking tablet 824bfd1d47374f10b3b63023d9b5e051
I20240503 02:11:47.769269   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 824bfd1d47374f10b3b63023d9b5e051
I20240503 02:11:47.769385   580 log_verifier.cc:126] Checking tablet 90223cd1477f46788581d2dcc937e162
I20240503 02:11:47.810915   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 90223cd1477f46788581d2dcc937e162
I20240503 02:11:47.811035   580 log_verifier.cc:126] Checking tablet 9095a299c3084a30b4ec750b5883ff9c
I20240503 02:11:47.851717   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9095a299c3084a30b4ec750b5883ff9c
I20240503 02:11:47.851857   580 log_verifier.cc:126] Checking tablet 9228c7369b9f4ddfb7a2bbf64b2afb02
I20240503 02:11:47.893765   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9228c7369b9f4ddfb7a2bbf64b2afb02
I20240503 02:11:47.893894   580 log_verifier.cc:126] Checking tablet 9a1a504c6d8e4bf89cc813b9cedbccd6
I20240503 02:11:47.934252   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9a1a504c6d8e4bf89cc813b9cedbccd6
I20240503 02:11:47.934393   580 log_verifier.cc:126] Checking tablet aaeb0eedcd0345a3aadb3f2fc9c484ba
I20240503 02:11:47.978271   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet aaeb0eedcd0345a3aadb3f2fc9c484ba
I20240503 02:11:47.978422   580 log_verifier.cc:126] Checking tablet ab506e9c0d6e42d69ce2cb7f4692e0b7
I20240503 02:11:48.022253   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ab506e9c0d6e42d69ce2cb7f4692e0b7
I20240503 02:11:48.022364   580 log_verifier.cc:126] Checking tablet b667f314948a4c2c917cc56dde38c67c
I20240503 02:11:48.065214   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b667f314948a4c2c917cc56dde38c67c
I20240503 02:11:48.065321   580 log_verifier.cc:126] Checking tablet c04e46c4e09740b2a847d140e0a56b12
I20240503 02:11:48.109009   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c04e46c4e09740b2a847d140e0a56b12
I20240503 02:11:48.109141   580 log_verifier.cc:126] Checking tablet c1d5289a99ca41dfba0ddaf9fbf6a72c
I20240503 02:11:48.151111   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c1d5289a99ca41dfba0ddaf9fbf6a72c
I20240503 02:11:48.151248   580 log_verifier.cc:126] Checking tablet c218fde67d2b41df8bd27d7cf4e1fe38
I20240503 02:11:48.194200   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c218fde67d2b41df8bd27d7cf4e1fe38
I20240503 02:11:48.194336   580 log_verifier.cc:126] Checking tablet c4ed1defa1314b86a5186c3349169694
I20240503 02:11:48.236605   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c4ed1defa1314b86a5186c3349169694
I20240503 02:11:48.236739   580 log_verifier.cc:126] Checking tablet ca2c4b67a0fe426e9fe8a9b00360b12c
I20240503 02:11:48.280102   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ca2c4b67a0fe426e9fe8a9b00360b12c
I20240503 02:11:48.280225   580 log_verifier.cc:126] Checking tablet d0e1479d6cb4465e8c29fb5c58415421
I20240503 02:11:48.321166   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d0e1479d6cb4465e8c29fb5c58415421
I20240503 02:11:48.321331   580 log_verifier.cc:126] Checking tablet d320d692be6c47b6b9ec43fe3dec719c
I20240503 02:11:48.363271   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d320d692be6c47b6b9ec43fe3dec719c
I20240503 02:11:48.363467   580 log_verifier.cc:126] Checking tablet d68da7e811f043dabeb7645bc3d8c4cd
I20240503 02:11:48.406256   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d68da7e811f043dabeb7645bc3d8c4cd
I20240503 02:11:48.406400   580 log_verifier.cc:126] Checking tablet d7f194bfa67c4aeea454483ecb13dcb3
I20240503 02:11:48.450421   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d7f194bfa67c4aeea454483ecb13dcb3
I20240503 02:11:48.450533   580 log_verifier.cc:126] Checking tablet e9eaf1bda2e047f1bdbf7783d0ebb989
I20240503 02:11:48.491843   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e9eaf1bda2e047f1bdbf7783d0ebb989
I20240503 02:11:48.491955   580 log_verifier.cc:126] Checking tablet f7335d586927431f8ccdbbea16a8decb
I20240503 02:11:48.533087   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f7335d586927431f8ccdbbea16a8decb
I20240503 02:11:48.533202   580 log_verifier.cc:126] Checking tablet fb165523c1e849eeb8656e2fc23b31dd
I20240503 02:11:48.581117   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fb165523c1e849eeb8656e2fc23b31dd
I20240503 02:11:48.581220   580 log_verifier.cc:126] Checking tablet ff7268bf9a9b47bca0168ec2e37122e8
I20240503 02:11:48.623610   580 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ff7268bf9a9b47bca0168ec2e37122e8
I20240503 02:11:48.643385   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 682
I20240503 02:11:48.649221   804 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:48.792076   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 682
W20240503 02:11:48.817206  1224 connection.cc:535] server connection from 127.0.145.1:33889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:11:48.817746   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 813
W20240503 02:11:48.817764  1223 connection.cc:535] client connection to 127.0.145.1:34401 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:11:48.818013  1223 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:11:48.820932  1223 consensus_peers.cc:485] T b667f314948a4c2c917cc56dde38c67c P 572709bd9ae041a4a98029298a6f69c1 -> Peer e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401): Couldn't send request to peer e8d9a43983f2478ea7ababa09b230177. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:34401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:48.824810   936 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:11:48.829237   961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.1:34401: connect: Connection refused (error 111)
W20240503 02:11:48.830802   961 consensus_peers.cc:485] T e9eaf1bda2e047f1bdbf7783d0ebb989 P f5e38dcb24084a1e969f83bf34aef88f -> Peer e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401): Couldn't send request to peer e8d9a43983f2478ea7ababa09b230177. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:34401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:48.895548   961 consensus_peers.cc:485] T ca2c4b67a0fe426e9fe8a9b00360b12c P f5e38dcb24084a1e969f83bf34aef88f -> Peer e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401): Couldn't send request to peer e8d9a43983f2478ea7ababa09b230177. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:34401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:48.966939   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 813
W20240503 02:11:48.984951  1223 consensus_peers.cc:485] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 -> Peer e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401): Couldn't send request to peer e8d9a43983f2478ea7ababa09b230177. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:34401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:48.992417  1355 connection.cc:535] client connection to 127.0.145.2:43485 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:11:48.992694  1355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:11:48.993055   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 944
W20240503 02:11:48.994930  1355 consensus_peers.cc:485] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e -> Peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485): Couldn't send request to peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:48.995064  1355 consensus_peers.cc:485] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e -> Peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485): Couldn't send request to peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:48.995186  1355 consensus_peers.cc:485] T 7e7f9eb8403e45d88a5532506d06e2e2 P 4fda3b525c234f8f88aaefd355e7274e -> Peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485): Couldn't send request to peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:48.995275  1355 consensus_peers.cc:485] T 6f6cdecb3c1d45dc8775f47bdd265d3c P 4fda3b525c234f8f88aaefd355e7274e -> Peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485): Couldn't send request to peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:49.000241  1066 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:11:49.001791  1223 consensus_peers.cc:485] T 18ccfc8dbf884ab3a3e3338d679787af P 572709bd9ae041a4a98029298a6f69c1 -> Peer e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401): Couldn't send request to peer e8d9a43983f2478ea7ababa09b230177. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:34401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:49.054549  1355 consensus_peers.cc:485] T d0e1479d6cb4465e8c29fb5c58415421 P 4fda3b525c234f8f88aaefd355e7274e -> Peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485): Couldn't send request to peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:49.093987  1093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.2:43485: connect: Connection refused (error 111)
W20240503 02:11:49.095667  1093 consensus_peers.cc:485] T 6947883c90ca46709a67ca185471178e P 288540692e7b404991570d9ca16ab6c0 -> Peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485): Couldn't send request to peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:49.135529   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 944
W20240503 02:11:49.138741  1223 consensus_peers.cc:485] T ab506e9c0d6e42d69ce2cb7f4692e0b7 P 572709bd9ae041a4a98029298a6f69c1 -> Peer e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401): Couldn't send request to peer e8d9a43983f2478ea7ababa09b230177. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:34401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:49.138986  1355 consensus_peers.cc:485] T c1d5289a99ca41dfba0ddaf9fbf6a72c P 4fda3b525c234f8f88aaefd355e7274e -> Peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485): Couldn't send request to peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:49.159518  1223 consensus_peers.cc:485] T aaeb0eedcd0345a3aadb3f2fc9c484ba P 572709bd9ae041a4a98029298a6f69c1 -> Peer e8d9a43983f2478ea7ababa09b230177 (127.0.145.1:34401): Couldn't send request to peer e8d9a43983f2478ea7ababa09b230177. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:34401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:49.162173  1222 connection.cc:535] server connection from 127.0.145.3:53585 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:11:49.162587   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 1075
I20240503 02:11:49.168412  1197 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:11:49.205650  1224 consensus_peers.cc:485] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 -> Peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485): Couldn't send request to peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:49.229710  1223 consensus_peers.cc:485] T fb165523c1e849eeb8656e2fc23b31dd P 572709bd9ae041a4a98029298a6f69c1 -> Peer f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511): Couldn't send request to peer f5e38dcb24084a1e969f83bf34aef88f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.3:35511: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:49.247385  1355 consensus_peers.cc:485] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e -> Peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485): Couldn't send request to peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:49.289793  1223 consensus_peers.cc:485] T d68da7e811f043dabeb7645bc3d8c4cd P 572709bd9ae041a4a98029298a6f69c1 -> Peer f5e38dcb24084a1e969f83bf34aef88f (127.0.145.3:35511): Couldn't send request to peer f5e38dcb24084a1e969f83bf34aef88f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.3:35511: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:49.292016  1355 consensus_peers.cc:485] T 824bfd1d47374f10b3b63023d9b5e051 P 4fda3b525c234f8f88aaefd355e7274e -> Peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485): Couldn't send request to peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:49.303350   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 1075
W20240503 02:11:49.311769  1355 consensus_peers.cc:485] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e -> Peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485): Couldn't send request to peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:49.328467   610 connection.cc:535] server connection from 127.0.145.4:34171 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:11:49.329015   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 1206
W20240503 02:11:49.340068  1355 consensus_peers.cc:485] T c218fde67d2b41df8bd27d7cf4e1fe38 P 4fda3b525c234f8f88aaefd355e7274e -> Peer 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773): Couldn't send request to peer 288540692e7b404991570d9ca16ab6c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.4:32773: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:49.341107  1328 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:11:49.342777  1355 consensus_peers.cc:485] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e -> Peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b (127.0.145.2:43485): Couldn't send request to peer 88791c3c9c9341a6a2d8a6b5d3aa1f1b. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:43485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:49.343705  1355 consensus_peers.cc:485] T c04e46c4e09740b2a847d140e0a56b12 P 4fda3b525c234f8f88aaefd355e7274e -> Peer 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773): Couldn't send request to peer 288540692e7b404991570d9ca16ab6c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.4:32773: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:49.358295  1355 consensus_peers.cc:485] T 9095a299c3084a30b4ec750b5883ff9c P 4fda3b525c234f8f88aaefd355e7274e -> Peer 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773): Couldn't send request to peer 288540692e7b404991570d9ca16ab6c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.4:32773: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:49.459419  1355 consensus_peers.cc:485] T 4775eb444281418e830ed6944f5f8ec3 P 4fda3b525c234f8f88aaefd355e7274e -> Peer 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773): Couldn't send request to peer 288540692e7b404991570d9ca16ab6c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.4:32773: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:49.478137   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 1206
W20240503 02:11:49.503448  1355 consensus_peers.cc:485] T 0751c6ca11054c0f93d45574c49dc1ac P 4fda3b525c234f8f88aaefd355e7274e -> Peer 288540692e7b404991570d9ca16ab6c0 (127.0.145.4:32773): Couldn't send request to peer 288540692e7b404991570d9ca16ab6c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.4:32773: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:49.503656   609 connection.cc:535] server connection from 127.0.145.5:39027 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:11:49.503885   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 1337
I20240503 02:11:49.509598  1460 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:49.645336   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 1337
I20240503 02:11:49.670120   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 592
I20240503 02:11:49.671231   653 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:49.822805   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 592
2024-05-03T02:11:49Z chronyd exiting
[       OK ] LocationAwareRebalancingBasicTest.Basic (32503 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (32503 ms total)

[----------] 1 test from RebalanceStartCriteriaTest
[ RUN      ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240503 02:11:49.982110   580 test_util.cc:268] Using random seed: 1041563511
I20240503 02:11:49.984239   580 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:11:49.984326   580 ts_itest-base.cc:113] --------------
I20240503 02:11:49.984364   580 ts_itest-base.cc:114] 5 tablet servers
I20240503 02:11:49.984438   580 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:11:49.984483   580 ts_itest-base.cc:116] --------------
2024-05-03T02:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:11:49Z Disabled control of system clock
I20240503 02:11:50.020262   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.145.62:34539
--webserver_interface=127.0.145.62
--webserver_port=0
--builtin_ntp_servers=127.0.145.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.145.62:34539
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 02:11:50.357650  2928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:50.357990  2928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:50.358083  2928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:50.365639  2928 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:50.365780  2928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:11:50.365823  2928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:50.365875  2928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:11:50.365918  2928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:11:50.375231  2928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.145.62:34539
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.145.62:34539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.145.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:50.376566  2928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:50.378293  2928 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:50.390249  2934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:50.391144  2936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:50.391808  2933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:50.392920  2928 server_base.cc:1034] running on GCE node
I20240503 02:11:50.393736  2928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:50.395538  2928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:50.396781  2928 hybrid_clock.cc:648] HybridClock initialized: now 1714702310396716 us; error 57 us; skew 500 ppm
I20240503 02:11:50.397166  2928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:50.399566  2928 webserver.cc:460] Webserver started at http://127.0.145.62:43749/ using document root <none> and password file <none>
I20240503 02:11:50.400313  2928 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:50.400437  2928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:50.400784  2928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:50.403484  2928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1d1ba5ac91c24cb08ebf760a5376f8cc"
format_stamp: "Formatted at 2024-05-03 02:11:50 on dist-test-slave-gq9h"
I20240503 02:11:50.404248  2928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1d1ba5ac91c24cb08ebf760a5376f8cc"
format_stamp: "Formatted at 2024-05-03 02:11:50 on dist-test-slave-gq9h"
I20240503 02:11:50.409497  2928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:11:50.413295  2942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:50.414777  2928 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240503 02:11:50.414952  2928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1d1ba5ac91c24cb08ebf760a5376f8cc"
format_stamp: "Formatted at 2024-05-03 02:11:50 on dist-test-slave-gq9h"
I20240503 02:11:50.415155  2928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:50.436955  2928 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:50.437826  2928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:50.438093  2928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:50.455538  2928 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.62:34539
I20240503 02:11:50.455562  2993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.62:34539 every 8 connection(s)
I20240503 02:11:50.456951  2928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:11:50.460793  2994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.464272   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 2928
I20240503 02:11:50.464530   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:11:50.471958  2994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc: Bootstrap starting.
I20240503 02:11:50.475695  2994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:50.476876  2994 log.cc:824] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:50.479722  2994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc: No bootstrap required, opened a new log
I20240503 02:11:50.485832  2994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1ba5ac91c24cb08ebf760a5376f8cc" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 34539 } }
I20240503 02:11:50.486217  2994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:50.486383  2994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1ba5ac91c24cb08ebf760a5376f8cc, State: Initialized, Role: FOLLOWER
I20240503 02:11:50.487067  2994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1ba5ac91c24cb08ebf760a5376f8cc" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 34539 } }
I20240503 02:11:50.487258  2994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:50.487391  2994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:50.487542  2994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:50.489535  2994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1ba5ac91c24cb08ebf760a5376f8cc" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 34539 } }
I20240503 02:11:50.490072  2994 leader_election.cc:302] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d1ba5ac91c24cb08ebf760a5376f8cc; no voters: 
I20240503 02:11:50.490773  2994 leader_election.cc:288] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:11:50.490888  2999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:50.491516  2999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [term 1 LEADER]: Becoming Leader. State: Replica: 1d1ba5ac91c24cb08ebf760a5376f8cc, State: Running, Role: LEADER
I20240503 02:11:50.492079  2999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1ba5ac91c24cb08ebf760a5376f8cc" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 34539 } }
I20240503 02:11:50.492740  2994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:50.494869  3001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d1ba5ac91c24cb08ebf760a5376f8cc. Latest consensus state: current_term: 1 leader_uuid: "1d1ba5ac91c24cb08ebf760a5376f8cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1ba5ac91c24cb08ebf760a5376f8cc" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 34539 } } }
I20240503 02:11:50.495209  3001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:50.496148  3000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d1ba5ac91c24cb08ebf760a5376f8cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1ba5ac91c24cb08ebf760a5376f8cc" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 34539 } } }
I20240503 02:11:50.496557  3000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:50.497948  3005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:50.503700  3005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:50.513346  3005 catalog_manager.cc:1347] Generated new cluster ID: e547893b3a0546a9af51c745280bbc23
I20240503 02:11:50.513496  3005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:50.541363  3005 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:50.543243  3005 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:50.556499  3005 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1d1ba5ac91c24cb08ebf760a5376f8cc: Generated new TSK 0
I20240503 02:11:50.557590  3005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:50.565188   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.1:0
--local_ip_for_outbound_sockets=127.0.145.1
--webserver_interface=127.0.145.1
--webserver_port=0
--tserver_master_addrs=127.0.145.62:34539
--builtin_ntp_servers=127.0.145.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:11:50.907196  3018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:50.907510  3018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:50.907650  3018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:50.915359  3018 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:50.915505  3018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:50.915663  3018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.1
I20240503 02:11:50.925212  3018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.145.1
--webserver_port=0
--tserver_master_addrs=127.0.145.62:34539
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.1
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:50.926581  3018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:50.928443  3018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:50.942126  3024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:50.942983  3023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:50.943907  3026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:50.944895  3018 server_base.cc:1034] running on GCE node
I20240503 02:11:50.945567  3018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:50.946579  3018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:50.947845  3018 hybrid_clock.cc:648] HybridClock initialized: now 1714702310947794 us; error 53 us; skew 500 ppm
I20240503 02:11:50.948170  3018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:50.950564  3018 webserver.cc:460] Webserver started at http://127.0.145.1:38451/ using document root <none> and password file <none>
I20240503 02:11:50.951372  3018 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:50.951503  3018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:50.951838  3018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:50.954665  3018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c445e7b697714817806277c74a8cb973"
format_stamp: "Formatted at 2024-05-03 02:11:50 on dist-test-slave-gq9h"
I20240503 02:11:50.955521  3018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c445e7b697714817806277c74a8cb973"
format_stamp: "Formatted at 2024-05-03 02:11:50 on dist-test-slave-gq9h"
I20240503 02:11:50.961110  3018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.004s
I20240503 02:11:50.964854  3032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:50.966293  3018 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:50.966461  3018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c445e7b697714817806277c74a8cb973"
format_stamp: "Formatted at 2024-05-03 02:11:50 on dist-test-slave-gq9h"
I20240503 02:11:50.966723  3018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:50.984913  3018 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:50.985785  3018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:50.986052  3018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:50.987069  3018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:50.988559  3018 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:50.988648  3018 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:50.988765  3018 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:50.988842  3018 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:51.021500  3018 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.1:40361
I20240503 02:11:51.021549  3144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.1:40361 every 8 connection(s)
I20240503 02:11:51.022889  3018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:11:51.028972   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 3018
I20240503 02:11:51.029147   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:11:51.032464   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.2:0
--local_ip_for_outbound_sockets=127.0.145.2
--webserver_interface=127.0.145.2
--webserver_port=0
--tserver_master_addrs=127.0.145.62:34539
--builtin_ntp_servers=127.0.145.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:51.036664  3145 heartbeater.cc:347] Connected to a master server at 127.0.145.62:34539
I20240503 02:11:51.036983  3145 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:51.037868  3145 heartbeater.cc:511] Master 127.0.145.62:34539 requested a full tablet report, sending...
I20240503 02:11:51.040117  2959 ts_manager.cc:194] Registered new tserver with Master: c445e7b697714817806277c74a8cb973 (127.0.145.1:40361)
I20240503 02:11:51.041651  2959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.1:36493
W20240503 02:11:51.392467  3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:51.392959  3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:51.393138  3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:51.403942  3149 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:51.404115  3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:51.404260  3149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.2
I20240503 02:11:51.413594  3149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.145.2
--webserver_port=0
--tserver_master_addrs=127.0.145.62:34539
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.2
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:51.415056  3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:51.416931  3149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:51.431824  3155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:51.432657  3154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:51.433570  3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:51.434423  3149 server_base.cc:1034] running on GCE node
I20240503 02:11:51.435248  3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:51.436337  3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:51.437561  3149 hybrid_clock.cc:648] HybridClock initialized: now 1714702311437509 us; error 48 us; skew 500 ppm
I20240503 02:11:51.437907  3149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:51.440375  3149 webserver.cc:460] Webserver started at http://127.0.145.2:45887/ using document root <none> and password file <none>
I20240503 02:11:51.441179  3149 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:51.441311  3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:51.441660  3149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:51.444494  3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8842a292818546f886ee9c780deeeeda"
format_stamp: "Formatted at 2024-05-03 02:11:51 on dist-test-slave-gq9h"
I20240503 02:11:51.445329  3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8842a292818546f886ee9c780deeeeda"
format_stamp: "Formatted at 2024-05-03 02:11:51 on dist-test-slave-gq9h"
I20240503 02:11:51.451086  3149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.004s
I20240503 02:11:51.454892  3163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:51.456612  3149 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:11:51.456816  3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8842a292818546f886ee9c780deeeeda"
format_stamp: "Formatted at 2024-05-03 02:11:51 on dist-test-slave-gq9h"
I20240503 02:11:51.457034  3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:51.473634  3149 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:51.474529  3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:51.474835  3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:51.475804  3149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:51.477353  3149 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:51.477456  3149 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:51.477578  3149 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:51.477655  3149 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:51.510849  3149 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.2:41635
I20240503 02:11:51.510887  3275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.2:41635 every 8 connection(s)
I20240503 02:11:51.512272  3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:11:51.517894   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 3149
I20240503 02:11:51.518095   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:11:51.521755   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.3:0
--local_ip_for_outbound_sockets=127.0.145.3
--webserver_interface=127.0.145.3
--webserver_port=0
--tserver_master_addrs=127.0.145.62:34539
--builtin_ntp_servers=127.0.145.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:51.525959  3276 heartbeater.cc:347] Connected to a master server at 127.0.145.62:34539
I20240503 02:11:51.526298  3276 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:51.527196  3276 heartbeater.cc:511] Master 127.0.145.62:34539 requested a full tablet report, sending...
I20240503 02:11:51.528713  2959 ts_manager.cc:194] Registered new tserver with Master: 8842a292818546f886ee9c780deeeeda (127.0.145.2:41635)
I20240503 02:11:51.529513  2959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.2:47989
W20240503 02:11:51.866237  3279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:51.866560  3279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:51.866716  3279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:51.874408  3279 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:51.874634  3279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:51.874802  3279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.3
I20240503 02:11:51.884260  3279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.145.3
--webserver_port=0
--tserver_master_addrs=127.0.145.62:34539
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.3
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:51.885713  3279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:51.887635  3279 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:51.900791  3285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:51.901692  3279 server_base.cc:1034] running on GCE node
W20240503 02:11:51.902454  3284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:51.903303  3287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:51.904012  3279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:51.904986  3279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:51.906188  3279 hybrid_clock.cc:648] HybridClock initialized: now 1714702311906125 us; error 57 us; skew 500 ppm
I20240503 02:11:51.906512  3279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:51.908900  3279 webserver.cc:460] Webserver started at http://127.0.145.3:33793/ using document root <none> and password file <none>
I20240503 02:11:51.909680  3279 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:51.909808  3279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:51.910135  3279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:51.913008  3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a10baccf03a04bd8bf886d6ccf54b2a2"
format_stamp: "Formatted at 2024-05-03 02:11:51 on dist-test-slave-gq9h"
I20240503 02:11:51.913812  3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a10baccf03a04bd8bf886d6ccf54b2a2"
format_stamp: "Formatted at 2024-05-03 02:11:51 on dist-test-slave-gq9h"
I20240503 02:11:51.919349  3279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240503 02:11:51.923028  3293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:51.924451  3279 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 02:11:51.924624  3279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a10baccf03a04bd8bf886d6ccf54b2a2"
format_stamp: "Formatted at 2024-05-03 02:11:51 on dist-test-slave-gq9h"
I20240503 02:11:51.924829  3279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:51.941537  3279 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:51.942451  3279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:51.942760  3279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:51.943769  3279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:51.945343  3279 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:51.945451  3279 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:51.945560  3279 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:51.945628  3279 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:51.982019  3279 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.3:33003
I20240503 02:11:51.982128  3405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.3:33003 every 8 connection(s)
I20240503 02:11:51.983525  3279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:51.986852   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 3279
I20240503 02:11:51.987010   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:11:51.990542   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.4:0
--local_ip_for_outbound_sockets=127.0.145.4
--webserver_interface=127.0.145.4
--webserver_port=0
--tserver_master_addrs=127.0.145.62:34539
--builtin_ntp_servers=127.0.145.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:51.998014  3406 heartbeater.cc:347] Connected to a master server at 127.0.145.62:34539
I20240503 02:11:51.998416  3406 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:51.999598  3406 heartbeater.cc:511] Master 127.0.145.62:34539 requested a full tablet report, sending...
I20240503 02:11:52.001232  2959 ts_manager.cc:194] Registered new tserver with Master: a10baccf03a04bd8bf886d6ccf54b2a2 (127.0.145.3:33003)
I20240503 02:11:52.002061  2959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.3:56387
I20240503 02:11:52.044942  3145 heartbeater.cc:503] Master 127.0.145.62:34539 was elected leader, sending a full tablet report...
W20240503 02:11:52.334969  3409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:52.335280  3409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:52.335431  3409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:52.343261  3409 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:52.343398  3409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:52.343577  3409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.4
I20240503 02:11:52.352857  3409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.145.4
--webserver_port=0
--tserver_master_addrs=127.0.145.62:34539
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.4
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:52.354238  3409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:52.356133  3409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:52.369037  3414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:52.369923  3417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:52.370561  3415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:52.371560  3409 server_base.cc:1034] running on GCE node
I20240503 02:11:52.372231  3409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:52.373234  3409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:52.374468  3409 hybrid_clock.cc:648] HybridClock initialized: now 1714702312374407 us; error 40 us; skew 500 ppm
I20240503 02:11:52.374871  3409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:52.377310  3409 webserver.cc:460] Webserver started at http://127.0.145.4:40347/ using document root <none> and password file <none>
I20240503 02:11:52.378104  3409 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:52.378232  3409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:52.378576  3409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:52.381544  3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "33e30e7a08e24a869354e7d1e40e78c4"
format_stamp: "Formatted at 2024-05-03 02:11:52 on dist-test-slave-gq9h"
I20240503 02:11:52.382442  3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "33e30e7a08e24a869354e7d1e40e78c4"
format_stamp: "Formatted at 2024-05-03 02:11:52 on dist-test-slave-gq9h"
I20240503 02:11:52.388473  3409 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.001s
I20240503 02:11:52.392632  3423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:52.394359  3409 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.000s
I20240503 02:11:52.394587  3409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "33e30e7a08e24a869354e7d1e40e78c4"
format_stamp: "Formatted at 2024-05-03 02:11:52 on dist-test-slave-gq9h"
I20240503 02:11:52.394881  3409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:52.422987  3409 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:52.423907  3409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:52.424196  3409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:52.425197  3409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:52.426831  3409 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:52.426949  3409 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:52.427086  3409 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:52.427170  3409 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:52.460095  3409 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.4:38347
I20240503 02:11:52.460165  3535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.4:38347 every 8 connection(s)
I20240503 02:11:52.461552  3409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:11:52.469475   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 3409
I20240503 02:11:52.469674   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:11:52.473169   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.5:0
--local_ip_for_outbound_sockets=127.0.145.5
--webserver_interface=127.0.145.5
--webserver_port=0
--tserver_master_addrs=127.0.145.62:34539
--builtin_ntp_servers=127.0.145.20:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:52.474982  3536 heartbeater.cc:347] Connected to a master server at 127.0.145.62:34539
I20240503 02:11:52.475286  3536 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:52.476122  3536 heartbeater.cc:511] Master 127.0.145.62:34539 requested a full tablet report, sending...
I20240503 02:11:52.477697  2959 ts_manager.cc:194] Registered new tserver with Master: 33e30e7a08e24a869354e7d1e40e78c4 (127.0.145.4:38347)
I20240503 02:11:52.478883  2959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.4:55227
I20240503 02:11:52.532088  3276 heartbeater.cc:503] Master 127.0.145.62:34539 was elected leader, sending a full tablet report...
W20240503 02:11:52.821561  3539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:52.821880  3539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:52.822016  3539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:52.829957  3539 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:52.830116  3539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:52.830263  3539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.5
I20240503 02:11:52.839702  3539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.145.5
--webserver_port=0
--tserver_master_addrs=127.0.145.62:34539
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.5
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:52.841027  3539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:52.842833  3539 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:52.855921  3544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:52.856807  3547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:52.857390  3545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:52.858402  3539 server_base.cc:1034] running on GCE node
I20240503 02:11:52.859063  3539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:52.860038  3539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:52.861239  3539 hybrid_clock.cc:648] HybridClock initialized: now 1714702312861178 us; error 56 us; skew 500 ppm
I20240503 02:11:52.861608  3539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:52.864250  3539 webserver.cc:460] Webserver started at http://127.0.145.5:35535/ using document root <none> and password file <none>
I20240503 02:11:52.865032  3539 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:52.865134  3539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:52.865429  3539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:52.868176  3539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe"
format_stamp: "Formatted at 2024-05-03 02:11:52 on dist-test-slave-gq9h"
I20240503 02:11:52.868971  3539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe"
format_stamp: "Formatted at 2024-05-03 02:11:52 on dist-test-slave-gq9h"
I20240503 02:11:52.874426  3539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240503 02:11:52.878264  3553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:52.879873  3539 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:11:52.880054  3539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe"
format_stamp: "Formatted at 2024-05-03 02:11:52 on dist-test-slave-gq9h"
I20240503 02:11:52.880265  3539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:52.899415  3539 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:52.900344  3539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:52.900630  3539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:52.901659  3539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:52.903301  3539 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:52.903409  3539 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:52.903527  3539 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:52.903601  3539 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:52.936321  3539 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.5:37653
I20240503 02:11:52.936362  3665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.5:37653 every 8 connection(s)
I20240503 02:11:52.937713  3539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:52.938879   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 3539
I20240503 02:11:52.939047   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:11:52.951478  3666 heartbeater.cc:347] Connected to a master server at 127.0.145.62:34539
I20240503 02:11:52.951812  3666 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:52.952703  3666 heartbeater.cc:511] Master 127.0.145.62:34539 requested a full tablet report, sending...
I20240503 02:11:52.954207  2959 ts_manager.cc:194] Registered new tserver with Master: b10ff7bbd3a74a9ca5f4a2a953504ebe (127.0.145.5:37653)
I20240503 02:11:52.955137  2959 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.5:35319
I20240503 02:11:52.955616   580 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:11:52.972930  2959 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32804:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240503 02:11:53.005416  3406 heartbeater.cc:503] Master 127.0.145.62:34539 was elected leader, sending a full tablet report...
I20240503 02:11:53.006032  3211 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6073923baa4a038a7092f559afaedf (DEFAULT_TABLE table=TestTable [id=2b729538214d4bb7bc7a103c65d8ba9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.006347  3601 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6073923baa4a038a7092f559afaedf (DEFAULT_TABLE table=TestTable [id=2b729538214d4bb7bc7a103c65d8ba9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.007648  3211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6073923baa4a038a7092f559afaedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.007855  3601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6073923baa4a038a7092f559afaedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.007980  3341 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6073923baa4a038a7092f559afaedf (DEFAULT_TABLE table=TestTable [id=2b729538214d4bb7bc7a103c65d8ba9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.009322  3341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6073923baa4a038a7092f559afaedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.019428  3684 tablet_bootstrap.cc:492] T 0f6073923baa4a038a7092f559afaedf P 8842a292818546f886ee9c780deeeeda: Bootstrap starting.
I20240503 02:11:53.021916  3685 tablet_bootstrap.cc:492] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe: Bootstrap starting.
I20240503 02:11:53.025260  3684 tablet_bootstrap.cc:654] T 0f6073923baa4a038a7092f559afaedf P 8842a292818546f886ee9c780deeeeda: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.025586  3685 tablet_bootstrap.cc:654] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.025640  3686 tablet_bootstrap.cc:492] T 0f6073923baa4a038a7092f559afaedf P a10baccf03a04bd8bf886d6ccf54b2a2: Bootstrap starting.
I20240503 02:11:53.026971  3685 log.cc:824] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:53.027252  3684 log.cc:824] T 0f6073923baa4a038a7092f559afaedf P 8842a292818546f886ee9c780deeeeda: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:53.028904  3686 tablet_bootstrap.cc:654] T 0f6073923baa4a038a7092f559afaedf P a10baccf03a04bd8bf886d6ccf54b2a2: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.029974  3685 tablet_bootstrap.cc:492] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe: No bootstrap required, opened a new log
I20240503 02:11:53.030087  3686 log.cc:824] T 0f6073923baa4a038a7092f559afaedf P a10baccf03a04bd8bf886d6ccf54b2a2: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:53.030251  3685 ts_tablet_manager.cc:1367] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240503 02:11:53.031522  3684 tablet_bootstrap.cc:492] T 0f6073923baa4a038a7092f559afaedf P 8842a292818546f886ee9c780deeeeda: No bootstrap required, opened a new log
I20240503 02:11:53.031886  3684 ts_tablet_manager.cc:1367] T 0f6073923baa4a038a7092f559afaedf P 8842a292818546f886ee9c780deeeeda: Time spent bootstrapping tablet: real 0.013s	user 0.004s	sys 0.004s
I20240503 02:11:53.037163  3685 raft_consensus.cc:348] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842a292818546f886ee9c780deeeeda" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41635 } } peers { permanent_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 37653 } } peers { permanent_uuid: "a10baccf03a04bd8bf886d6ccf54b2a2" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 33003 } }
I20240503 02:11:53.037519  3686 tablet_bootstrap.cc:492] T 0f6073923baa4a038a7092f559afaedf P a10baccf03a04bd8bf886d6ccf54b2a2: No bootstrap required, opened a new log
I20240503 02:11:53.037561  3685 raft_consensus.cc:374] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.037660  3685 raft_consensus.cc:729] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b10ff7bbd3a74a9ca5f4a2a953504ebe, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.037869  3686 ts_tablet_manager.cc:1367] T 0f6073923baa4a038a7092f559afaedf P a10baccf03a04bd8bf886d6ccf54b2a2: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.007s
I20240503 02:11:53.038409  3685 consensus_queue.cc:260] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842a292818546f886ee9c780deeeeda" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41635 } } peers { permanent_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 37653 } } peers { permanent_uuid: "a10baccf03a04bd8bf886d6ccf54b2a2" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 33003 } }
I20240503 02:11:53.039604  3666 heartbeater.cc:503] Master 127.0.145.62:34539 was elected leader, sending a full tablet report...
I20240503 02:11:53.040280  3685 ts_tablet_manager.cc:1398] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.000s
I20240503 02:11:53.041402  3684 raft_consensus.cc:348] T 0f6073923baa4a038a7092f559afaedf P 8842a292818546f886ee9c780deeeeda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842a292818546f886ee9c780deeeeda" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41635 } } peers { permanent_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 37653 } } peers { permanent_uuid: "a10baccf03a04bd8bf886d6ccf54b2a2" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 33003 } }
I20240503 02:11:53.041869  3684 raft_consensus.cc:374] T 0f6073923baa4a038a7092f559afaedf P 8842a292818546f886ee9c780deeeeda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.041975  3684 raft_consensus.cc:729] T 0f6073923baa4a038a7092f559afaedf P 8842a292818546f886ee9c780deeeeda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8842a292818546f886ee9c780deeeeda, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.042722  3684 consensus_queue.cc:260] T 0f6073923baa4a038a7092f559afaedf P 8842a292818546f886ee9c780deeeeda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842a292818546f886ee9c780deeeeda" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41635 } } peers { permanent_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 37653 } } peers { permanent_uuid: "a10baccf03a04bd8bf886d6ccf54b2a2" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 33003 } }
I20240503 02:11:53.044596  3684 ts_tablet_manager.cc:1398] T 0f6073923baa4a038a7092f559afaedf P 8842a292818546f886ee9c780deeeeda: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.004s
I20240503 02:11:53.046581  3686 raft_consensus.cc:348] T 0f6073923baa4a038a7092f559afaedf P a10baccf03a04bd8bf886d6ccf54b2a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842a292818546f886ee9c780deeeeda" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41635 } } peers { permanent_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 37653 } } peers { permanent_uuid: "a10baccf03a04bd8bf886d6ccf54b2a2" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 33003 } }
I20240503 02:11:53.047117  3686 raft_consensus.cc:374] T 0f6073923baa4a038a7092f559afaedf P a10baccf03a04bd8bf886d6ccf54b2a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.047430  3686 raft_consensus.cc:729] T 0f6073923baa4a038a7092f559afaedf P a10baccf03a04bd8bf886d6ccf54b2a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10baccf03a04bd8bf886d6ccf54b2a2, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.048134  3686 consensus_queue.cc:260] T 0f6073923baa4a038a7092f559afaedf P a10baccf03a04bd8bf886d6ccf54b2a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842a292818546f886ee9c780deeeeda" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41635 } } peers { permanent_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 37653 } } peers { permanent_uuid: "a10baccf03a04bd8bf886d6ccf54b2a2" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 33003 } }
I20240503 02:11:53.049913  3686 ts_tablet_manager.cc:1398] T 0f6073923baa4a038a7092f559afaedf P a10baccf03a04bd8bf886d6ccf54b2a2: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.005s
I20240503 02:11:53.105638  3690 raft_consensus.cc:482] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:53.105979  3690 raft_consensus.cc:504] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842a292818546f886ee9c780deeeeda" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41635 } } peers { permanent_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 37653 } } peers { permanent_uuid: "a10baccf03a04bd8bf886d6ccf54b2a2" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 33003 } }
I20240503 02:11:53.108498  3690 leader_election.cc:288] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8842a292818546f886ee9c780deeeeda (127.0.145.2:41635), a10baccf03a04bd8bf886d6ccf54b2a2 (127.0.145.3:33003)
I20240503 02:11:53.116528  3231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6073923baa4a038a7092f559afaedf" candidate_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8842a292818546f886ee9c780deeeeda" is_pre_election: true
I20240503 02:11:53.116613  3361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6073923baa4a038a7092f559afaedf" candidate_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10baccf03a04bd8bf886d6ccf54b2a2" is_pre_election: true
I20240503 02:11:53.117045  3231 raft_consensus.cc:2454] T 0f6073923baa4a038a7092f559afaedf P 8842a292818546f886ee9c780deeeeda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b10ff7bbd3a74a9ca5f4a2a953504ebe in term 0.
I20240503 02:11:53.117069  3361 raft_consensus.cc:2454] T 0f6073923baa4a038a7092f559afaedf P a10baccf03a04bd8bf886d6ccf54b2a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b10ff7bbd3a74a9ca5f4a2a953504ebe in term 0.
I20240503 02:11:53.117708  3555 leader_election.cc:302] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a10baccf03a04bd8bf886d6ccf54b2a2, b10ff7bbd3a74a9ca5f4a2a953504ebe; no voters: 
I20240503 02:11:53.118124  3690 raft_consensus.cc:2781] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:53.118269  3690 raft_consensus.cc:482] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:53.118369  3690 raft_consensus.cc:3037] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.120617  3690 raft_consensus.cc:504] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842a292818546f886ee9c780deeeeda" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41635 } } peers { permanent_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 37653 } } peers { permanent_uuid: "a10baccf03a04bd8bf886d6ccf54b2a2" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 33003 } }
I20240503 02:11:53.121268  3690 leader_election.cc:288] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [CANDIDATE]: Term 1 election: Requested vote from peers 8842a292818546f886ee9c780deeeeda (127.0.145.2:41635), a10baccf03a04bd8bf886d6ccf54b2a2 (127.0.145.3:33003)
I20240503 02:11:53.121834  3231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6073923baa4a038a7092f559afaedf" candidate_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8842a292818546f886ee9c780deeeeda"
I20240503 02:11:53.122048  3231 raft_consensus.cc:3037] T 0f6073923baa4a038a7092f559afaedf P 8842a292818546f886ee9c780deeeeda [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.122069  3361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6073923baa4a038a7092f559afaedf" candidate_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10baccf03a04bd8bf886d6ccf54b2a2"
I20240503 02:11:53.122295  3361 raft_consensus.cc:3037] T 0f6073923baa4a038a7092f559afaedf P a10baccf03a04bd8bf886d6ccf54b2a2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.124346  3231 raft_consensus.cc:2454] T 0f6073923baa4a038a7092f559afaedf P 8842a292818546f886ee9c780deeeeda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b10ff7bbd3a74a9ca5f4a2a953504ebe in term 1.
I20240503 02:11:53.124495  3361 raft_consensus.cc:2454] T 0f6073923baa4a038a7092f559afaedf P a10baccf03a04bd8bf886d6ccf54b2a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b10ff7bbd3a74a9ca5f4a2a953504ebe in term 1.
I20240503 02:11:53.124866  3557 leader_election.cc:302] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8842a292818546f886ee9c780deeeeda, b10ff7bbd3a74a9ca5f4a2a953504ebe; no voters: 
I20240503 02:11:53.125182  3690 raft_consensus.cc:2781] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:53.125633  3690 raft_consensus.cc:686] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [term 1 LEADER]: Becoming Leader. State: Replica: b10ff7bbd3a74a9ca5f4a2a953504ebe, State: Running, Role: LEADER
I20240503 02:11:53.126197  3690 consensus_queue.cc:237] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842a292818546f886ee9c780deeeeda" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41635 } } peers { permanent_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 37653 } } peers { permanent_uuid: "a10baccf03a04bd8bf886d6ccf54b2a2" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 33003 } }
I20240503 02:11:53.131675  2959 catalog_manager.cc:5484] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe reported cstate change: term changed from 0 to 1, leader changed from <none> to b10ff7bbd3a74a9ca5f4a2a953504ebe (127.0.145.5). New cstate: current_term: 1 leader_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8842a292818546f886ee9c780deeeeda" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b10ff7bbd3a74a9ca5f4a2a953504ebe" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 37653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a10baccf03a04bd8bf886d6ccf54b2a2" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 33003 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:53.162664   580 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:11:53.164942   580 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c445e7b697714817806277c74a8cb973 to finish bootstrapping
I20240503 02:11:53.174578   580 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8842a292818546f886ee9c780deeeeda to finish bootstrapping
I20240503 02:11:53.182663   580 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a10baccf03a04bd8bf886d6ccf54b2a2 to finish bootstrapping
W20240503 02:11:53.189517  3667 tablet.cc:2369] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:53.191053   580 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 33e30e7a08e24a869354e7d1e40e78c4 to finish bootstrapping
I20240503 02:11:53.200381   580 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b10ff7bbd3a74a9ca5f4a2a953504ebe to finish bootstrapping
W20240503 02:11:53.236027  3407 tablet.cc:2369] T 0f6073923baa4a038a7092f559afaedf P a10baccf03a04bd8bf886d6ccf54b2a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:11:53.265239  3277 tablet.cc:2369] T 0f6073923baa4a038a7092f559afaedf P 8842a292818546f886ee9c780deeeeda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:53.481606  3536 heartbeater.cc:503] Master 127.0.145.62:34539 was elected leader, sending a full tablet report...
I20240503 02:11:53.606974  3690 consensus_queue.cc:1035] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10baccf03a04bd8bf886d6ccf54b2a2" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 33003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:53.619804  3694 consensus_queue.cc:1035] T 0f6073923baa4a038a7092f559afaedf P b10ff7bbd3a74a9ca5f4a2a953504ebe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8842a292818546f886ee9c780deeeeda" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:53.697335  3211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:53.698944  3471 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:53.706629  3341 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:53.707340  3601 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:53.708554  3080 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:53.875020  3471 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:53.875973  3211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:53.876803  3341 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:53.884558  3601 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:53.899646  3080 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.540997  3471 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.543354  3341 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.551294  3211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.563513  3601 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:54.564400  3080 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.725647  3601 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:54.727909  3341 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.730034  3080 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.731155  3471 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.739984  3211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.909777  3211 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.913147  3471 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.922677  3601 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:54.929131  3080 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:54.932166  3341 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:55.089792   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 3018
I20240503 02:11:55.091344  3139 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:55.183720   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 3018
I20240503 02:11:55.203500   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 3149
I20240503 02:11:55.204603  3270 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:55.299088   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 3149
I20240503 02:11:55.318169   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 3279
I20240503 02:11:55.319244  3400 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:55.415344   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 3279
I20240503 02:11:55.435791   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 3409
I20240503 02:11:55.436861  3531 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:55.528548   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 3409
W20240503 02:11:55.542194  2943 connection.cc:535] server connection from 127.0.145.4:55227 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:11:55.542445   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 3539
I20240503 02:11:55.543563  3661 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:55.641034   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 3539
I20240503 02:11:55.659221   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 2928
I20240503 02:11:55.660357  2989 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:55.773743   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 2928
2024-05-03T02:11:55Z chronyd exiting
[       OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (5836 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5836 ms total)

[----------] 1 test from RebalanceParamTest
[ RUN      ] RebalanceParamTest.Rebalance/3
I20240503 02:11:55.818833   580 test_util.cc:268] Using random seed: 1047400250
I20240503 02:11:55.820863   580 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:11:55.820945   580 ts_itest-base.cc:113] --------------
I20240503 02:11:55.821027   580 ts_itest-base.cc:114] 7 tablet servers
I20240503 02:11:55.821077   580 ts_itest-base.cc:115] 2 replicas per TS
I20240503 02:11:55.821146   580 ts_itest-base.cc:116] --------------
2024-05-03T02:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:11:55Z Disabled control of system clock
I20240503 02:11:55.856745   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.145.62:40001
--webserver_interface=127.0.145.62
--webserver_port=0
--builtin_ntp_servers=127.0.145.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.145.62:40001
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 02:11:56.196830  3891 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 02:11:56.197181  3891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:56.197299  3891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:56.197403  3891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:56.205524  3891 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:56.205664  3891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:11:56.205740  3891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:56.205816  3891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:11:56.205897  3891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:11:56.215510  3891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.145.62:40001
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.145.62:40001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.145.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:56.216900  3891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:56.218916  3891 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:56.230580  3896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:56.231189  3899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:56.232146  3897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:56.232971  3891 server_base.cc:1034] running on GCE node
I20240503 02:11:56.233815  3891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:56.235401  3891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:56.236624  3891 hybrid_clock.cc:648] HybridClock initialized: now 1714702316236555 us; error 43 us; skew 500 ppm
I20240503 02:11:56.236965  3891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:56.239501  3891 webserver.cc:460] Webserver started at http://127.0.145.62:45437/ using document root <none> and password file <none>
I20240503 02:11:56.240233  3891 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:56.240362  3891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:56.240689  3891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:56.243470  3891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1a0bcd52c0054aa6a0012cbd524d103c"
format_stamp: "Formatted at 2024-05-03 02:11:56 on dist-test-slave-gq9h"
I20240503 02:11:56.244238  3891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1a0bcd52c0054aa6a0012cbd524d103c"
format_stamp: "Formatted at 2024-05-03 02:11:56 on dist-test-slave-gq9h"
I20240503 02:11:56.249845  3891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240503 02:11:56.253479  3905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:56.255093  3891 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:11:56.255282  3891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1a0bcd52c0054aa6a0012cbd524d103c"
format_stamp: "Formatted at 2024-05-03 02:11:56 on dist-test-slave-gq9h"
I20240503 02:11:56.255484  3891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:56.278752  3891 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:56.279702  3891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:56.280026  3891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:56.298712  3891 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.62:40001
I20240503 02:11:56.298786  3956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.62:40001 every 8 connection(s)
I20240503 02:11:56.300143  3891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:11:56.304075  3957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:56.310400   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 3891
I20240503 02:11:56.310633   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:11:56.314107  3957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c: Bootstrap starting.
I20240503 02:11:56.318315  3957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:56.319862  3957 log.cc:824] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:56.322885  3957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c: No bootstrap required, opened a new log
I20240503 02:11:56.329775  3957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0bcd52c0054aa6a0012cbd524d103c" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 40001 } }
I20240503 02:11:56.330148  3957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:56.330256  3957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a0bcd52c0054aa6a0012cbd524d103c, State: Initialized, Role: FOLLOWER
I20240503 02:11:56.330870  3957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0bcd52c0054aa6a0012cbd524d103c" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 40001 } }
I20240503 02:11:56.331086  3957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:56.331207  3957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:56.331357  3957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:56.333563  3957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0bcd52c0054aa6a0012cbd524d103c" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 40001 } }
I20240503 02:11:56.334035  3957 leader_election.cc:302] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a0bcd52c0054aa6a0012cbd524d103c; no voters: 
I20240503 02:11:56.334681  3957 leader_election.cc:288] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:11:56.334868  3962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:56.335494  3962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [term 1 LEADER]: Becoming Leader. State: Replica: 1a0bcd52c0054aa6a0012cbd524d103c, State: Running, Role: LEADER
I20240503 02:11:56.335992  3962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0bcd52c0054aa6a0012cbd524d103c" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 40001 } }
I20240503 02:11:56.336656  3957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:56.340003  3963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a0bcd52c0054aa6a0012cbd524d103c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0bcd52c0054aa6a0012cbd524d103c" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 40001 } } }
I20240503 02:11:56.340431  3963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:56.341116  3964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a0bcd52c0054aa6a0012cbd524d103c. Latest consensus state: current_term: 1 leader_uuid: "1a0bcd52c0054aa6a0012cbd524d103c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0bcd52c0054aa6a0012cbd524d103c" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 40001 } } }
I20240503 02:11:56.341396  3964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:56.342099  3971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:56.347828  3971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:56.356592  3971 catalog_manager.cc:1347] Generated new cluster ID: 431139aa51ae4b5bbcdccbab0f4c52d3
I20240503 02:11:56.356720  3971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:56.366459  3971 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:56.367805  3971 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:56.384012  3971 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1a0bcd52c0054aa6a0012cbd524d103c: Generated new TSK 0
I20240503 02:11:56.385213  3971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:56.398893   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.1:0
--local_ip_for_outbound_sockets=127.0.145.1
--webserver_interface=127.0.145.1
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--builtin_ntp_servers=127.0.145.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:11:56.747020  3981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:56.747323  3981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:56.747437  3981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:56.755008  3981 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:56.755147  3981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:56.755270  3981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.1
I20240503 02:11:56.764606  3981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.145.1
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.1
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:56.765936  3981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:56.767714  3981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:56.781579  3986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:56.782325  3987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:56.783129  3989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:56.783998  3981 server_base.cc:1034] running on GCE node
I20240503 02:11:56.784641  3981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:56.785619  3981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:56.786814  3981 hybrid_clock.cc:648] HybridClock initialized: now 1714702316786762 us; error 61 us; skew 500 ppm
I20240503 02:11:56.787176  3981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:56.789551  3981 webserver.cc:460] Webserver started at http://127.0.145.1:46803/ using document root <none> and password file <none>
I20240503 02:11:56.790352  3981 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:56.790465  3981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:56.790830  3981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:56.793659  3981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c4af56cee8e546c384cf20f40deeacd7"
format_stamp: "Formatted at 2024-05-03 02:11:56 on dist-test-slave-gq9h"
I20240503 02:11:56.794497  3981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c4af56cee8e546c384cf20f40deeacd7"
format_stamp: "Formatted at 2024-05-03 02:11:56 on dist-test-slave-gq9h"
I20240503 02:11:56.799976  3981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.001s
I20240503 02:11:56.803596  3995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:56.805084  3981 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 02:11:56.805260  3981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c4af56cee8e546c384cf20f40deeacd7"
format_stamp: "Formatted at 2024-05-03 02:11:56 on dist-test-slave-gq9h"
I20240503 02:11:56.805473  3981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:56.819989  3981 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:56.820899  3981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:56.821172  3981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:56.822111  3981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:56.823731  3981 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:56.823834  3981 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:56.823961  3981 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:56.824033  3981 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:56.856714  3981 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.1:43835
I20240503 02:11:56.856765  4107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.1:43835 every 8 connection(s)
I20240503 02:11:56.858063  3981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:11:56.863427   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 3981
I20240503 02:11:56.863641   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:11:56.867141   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.2:0
--local_ip_for_outbound_sockets=127.0.145.2
--webserver_interface=127.0.145.2
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--builtin_ntp_servers=127.0.145.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:56.872030  4108 heartbeater.cc:347] Connected to a master server at 127.0.145.62:40001
I20240503 02:11:56.872411  4108 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:56.873499  4108 heartbeater.cc:511] Master 127.0.145.62:40001 requested a full tablet report, sending...
I20240503 02:11:56.876209  3922 ts_manager.cc:194] Registered new tserver with Master: c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:11:56.878464  3922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.1:45305
W20240503 02:11:57.211336  4112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:57.211757  4112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:57.211879  4112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:57.219612  4112 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:57.219748  4112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:57.219915  4112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.2
I20240503 02:11:57.229184  4112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.145.2
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.2
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:57.230523  4112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:57.232403  4112 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:57.245285  4117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:57.245929  4118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:57.246842  4120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:57.247663  4112 server_base.cc:1034] running on GCE node
I20240503 02:11:57.248363  4112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:57.249356  4112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:57.250576  4112 hybrid_clock.cc:648] HybridClock initialized: now 1714702317250528 us; error 40 us; skew 500 ppm
I20240503 02:11:57.250957  4112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:57.253718  4112 webserver.cc:460] Webserver started at http://127.0.145.2:32861/ using document root <none> and password file <none>
I20240503 02:11:57.254526  4112 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:57.254678  4112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:57.254988  4112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:57.257705  4112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d7f19d54f56849f495dc547de6a47ad2"
format_stamp: "Formatted at 2024-05-03 02:11:57 on dist-test-slave-gq9h"
I20240503 02:11:57.258443  4112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d7f19d54f56849f495dc547de6a47ad2"
format_stamp: "Formatted at 2024-05-03 02:11:57 on dist-test-slave-gq9h"
I20240503 02:11:57.264370  4112 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.003s	sys 0.001s
I20240503 02:11:57.267947  4126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:57.269382  4112 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:11:57.269562  4112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d7f19d54f56849f495dc547de6a47ad2"
format_stamp: "Formatted at 2024-05-03 02:11:57 on dist-test-slave-gq9h"
I20240503 02:11:57.269775  4112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:57.332759  4112 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:57.333663  4112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:57.333947  4112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:57.335014  4112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:57.336575  4112 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:57.336675  4112 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:57.336787  4112 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:57.336865  4112 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:57.368865  4112 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.2:44433
I20240503 02:11:57.368906  4238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.2:44433 every 8 connection(s)
I20240503 02:11:57.370260  4112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:11:57.374859   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 4112
I20240503 02:11:57.375057   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:11:57.378170   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.3:0
--local_ip_for_outbound_sockets=127.0.145.3
--webserver_interface=127.0.145.3
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--builtin_ntp_servers=127.0.145.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:57.384514  4239 heartbeater.cc:347] Connected to a master server at 127.0.145.62:40001
I20240503 02:11:57.384842  4239 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:57.385695  4239 heartbeater.cc:511] Master 127.0.145.62:40001 requested a full tablet report, sending...
I20240503 02:11:57.387411  3922 ts_manager.cc:194] Registered new tserver with Master: d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:11:57.388612  3922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.2:55793
W20240503 02:11:57.728792  4243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:57.729130  4243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:57.729238  4243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:57.737147  4243 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:57.737301  4243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:57.737435  4243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.3
I20240503 02:11:57.747249  4243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.145.3
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.3
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:57.748633  4243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:57.750479  4243 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:57.764343  4248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:57.765105  4249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:57.765954  4251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:57.767007  4243 server_base.cc:1034] running on GCE node
I20240503 02:11:57.767727  4243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:57.768822  4243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:57.770062  4243 hybrid_clock.cc:648] HybridClock initialized: now 1714702317770019 us; error 29 us; skew 500 ppm
I20240503 02:11:57.770455  4243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:57.772986  4243 webserver.cc:460] Webserver started at http://127.0.145.3:35827/ using document root <none> and password file <none>
I20240503 02:11:57.773845  4243 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:57.773974  4243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:57.774367  4243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:57.777456  4243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8b812251ceb44fbe8a54fea04a7f8dfc"
format_stamp: "Formatted at 2024-05-03 02:11:57 on dist-test-slave-gq9h"
I20240503 02:11:57.778311  4243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8b812251ceb44fbe8a54fea04a7f8dfc"
format_stamp: "Formatted at 2024-05-03 02:11:57 on dist-test-slave-gq9h"
I20240503 02:11:57.783875  4243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:11:57.787415  4257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:57.788828  4243 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:11:57.789022  4243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8b812251ceb44fbe8a54fea04a7f8dfc"
format_stamp: "Formatted at 2024-05-03 02:11:57 on dist-test-slave-gq9h"
I20240503 02:11:57.789250  4243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:57.810953  4243 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:57.811911  4243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:57.812199  4243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:57.813189  4243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:57.814894  4243 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:57.814998  4243 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:57.815121  4243 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:57.815192  4243 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:57.847795  4243 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.3:41599
I20240503 02:11:57.847841  4369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.3:41599 every 8 connection(s)
I20240503 02:11:57.849205  4243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:57.854882   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 4243
I20240503 02:11:57.855110   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:11:57.858795   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.4:0
--local_ip_for_outbound_sockets=127.0.145.4
--webserver_interface=127.0.145.4
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--builtin_ntp_servers=127.0.145.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:57.864485  4370 heartbeater.cc:347] Connected to a master server at 127.0.145.62:40001
I20240503 02:11:57.864856  4370 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:57.865932  4370 heartbeater.cc:511] Master 127.0.145.62:40001 requested a full tablet report, sending...
I20240503 02:11:57.867601  3922 ts_manager.cc:194] Registered new tserver with Master: 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:11:57.868378  3922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.3:55565
I20240503 02:11:57.881938  4108 heartbeater.cc:503] Master 127.0.145.62:40001 was elected leader, sending a full tablet report...
W20240503 02:11:58.209709  4373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:58.210062  4373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:58.210219  4373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:58.218010  4373 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:58.218183  4373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:58.218358  4373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.4
I20240503 02:11:58.227867  4373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.145.4
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.4
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:58.229267  4373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:58.231154  4373 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:58.244565  4378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:58.245323  4381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:58.246001  4379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:58.247089  4373 server_base.cc:1034] running on GCE node
I20240503 02:11:58.247807  4373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:58.248916  4373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:58.250169  4373 hybrid_clock.cc:648] HybridClock initialized: now 1714702318250112 us; error 56 us; skew 500 ppm
I20240503 02:11:58.250560  4373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:58.253083  4373 webserver.cc:460] Webserver started at http://127.0.145.4:34341/ using document root <none> and password file <none>
I20240503 02:11:58.253890  4373 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:58.254009  4373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:58.254360  4373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:58.257292  4373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "6ebeaff495e042c78067c5b18600dc98"
format_stamp: "Formatted at 2024-05-03 02:11:58 on dist-test-slave-gq9h"
I20240503 02:11:58.258132  4373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "6ebeaff495e042c78067c5b18600dc98"
format_stamp: "Formatted at 2024-05-03 02:11:58 on dist-test-slave-gq9h"
I20240503 02:11:58.263914  4373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.005s
I20240503 02:11:58.267627  4387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:58.269057  4373 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 02:11:58.269237  4373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6ebeaff495e042c78067c5b18600dc98"
format_stamp: "Formatted at 2024-05-03 02:11:58 on dist-test-slave-gq9h"
I20240503 02:11:58.269461  4373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:58.281915  4373 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:58.283028  4373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:58.283341  4373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:58.284319  4373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:58.286078  4373 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:58.286185  4373 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:58.286314  4373 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:58.286387  4373 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:58.320348  4373 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.4:33659
I20240503 02:11:58.320367  4499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.4:33659 every 8 connection(s)
I20240503 02:11:58.321730  4373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:11:58.326102   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 4373
I20240503 02:11:58.326261   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:11:58.329351   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.5:0
--local_ip_for_outbound_sockets=127.0.145.5
--webserver_interface=127.0.145.5
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--builtin_ntp_servers=127.0.145.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:58.335284  4500 heartbeater.cc:347] Connected to a master server at 127.0.145.62:40001
I20240503 02:11:58.335608  4500 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:58.336546  4500 heartbeater.cc:511] Master 127.0.145.62:40001 requested a full tablet report, sending...
I20240503 02:11:58.338178  3922 ts_manager.cc:194] Registered new tserver with Master: 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4:33659)
I20240503 02:11:58.339025  3922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.4:54981
I20240503 02:11:58.391959  4239 heartbeater.cc:503] Master 127.0.145.62:40001 was elected leader, sending a full tablet report...
W20240503 02:11:58.682138  4503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:58.682462  4503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:58.682564  4503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:58.690212  4503 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:58.690364  4503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:58.690483  4503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.5
I20240503 02:11:58.699973  4503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.145.5
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.5
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:58.701349  4503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:58.703218  4503 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:58.716814  4511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:58.717670  4509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:58.718531  4508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:58.719584  4503 server_base.cc:1034] running on GCE node
I20240503 02:11:58.720304  4503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:58.721388  4503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:58.722644  4503 hybrid_clock.cc:648] HybridClock initialized: now 1714702318722558 us; error 45 us; skew 500 ppm
I20240503 02:11:58.723009  4503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:58.725522  4503 webserver.cc:460] Webserver started at http://127.0.145.5:43361/ using document root <none> and password file <none>
I20240503 02:11:58.726378  4503 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:58.726509  4503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:58.726887  4503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:58.729732  4503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "37c60eb66a6b45aaac4e0d7061572da6"
format_stamp: "Formatted at 2024-05-03 02:11:58 on dist-test-slave-gq9h"
I20240503 02:11:58.730584  4503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "37c60eb66a6b45aaac4e0d7061572da6"
format_stamp: "Formatted at 2024-05-03 02:11:58 on dist-test-slave-gq9h"
I20240503 02:11:58.736294  4503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.000s	sys 0.004s
I20240503 02:11:58.740293  4517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:58.741896  4503 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:11:58.742090  4503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "37c60eb66a6b45aaac4e0d7061572da6"
format_stamp: "Formatted at 2024-05-03 02:11:58 on dist-test-slave-gq9h"
I20240503 02:11:58.742309  4503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:58.768141  4503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:58.769057  4503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:58.769351  4503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:58.770421  4503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:58.772040  4503 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:58.772148  4503 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:58.772269  4503 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:58.772342  4503 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:58.804260  4503 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.5:39125
I20240503 02:11:58.804311  4629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.5:39125 every 8 connection(s)
I20240503 02:11:58.805670  4503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:58.816071   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 4503
I20240503 02:11:58.816277   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:11:58.818913  4630 heartbeater.cc:347] Connected to a master server at 127.0.145.62:40001
I20240503 02:11:58.819227  4630 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:58.819411   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.6:0
--local_ip_for_outbound_sockets=127.0.145.6
--webserver_interface=127.0.145.6
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--builtin_ntp_servers=127.0.145.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:58.820129  4630 heartbeater.cc:511] Master 127.0.145.62:40001 requested a full tablet report, sending...
I20240503 02:11:58.821636  3921 ts_manager.cc:194] Registered new tserver with Master: 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5:39125)
I20240503 02:11:58.822496  3921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.5:33819
I20240503 02:11:58.871129  4370 heartbeater.cc:503] Master 127.0.145.62:40001 was elected leader, sending a full tablet report...
W20240503 02:11:59.175333  4633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:59.175654  4633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:59.175805  4633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:59.184245  4633 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:59.184387  4633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:59.184569  4633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.6
I20240503 02:11:59.194000  4633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.145.6
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.6
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:59.195483  4633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:59.197361  4633 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:59.210285  4638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:59.210867  4639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:59.211763  4641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:59.212554  4633 server_base.cc:1034] running on GCE node
I20240503 02:11:59.213207  4633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:59.214228  4633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:59.215432  4633 hybrid_clock.cc:648] HybridClock initialized: now 1714702319215385 us; error 49 us; skew 500 ppm
I20240503 02:11:59.215770  4633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:59.218129  4633 webserver.cc:460] Webserver started at http://127.0.145.6:41177/ using document root <none> and password file <none>
I20240503 02:11:59.218984  4633 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:59.219105  4633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:59.219425  4633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:59.222124  4633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9c804fe67ee34e7fae5fab23db90f298"
format_stamp: "Formatted at 2024-05-03 02:11:59 on dist-test-slave-gq9h"
I20240503 02:11:59.222939  4633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9c804fe67ee34e7fae5fab23db90f298"
format_stamp: "Formatted at 2024-05-03 02:11:59 on dist-test-slave-gq9h"
I20240503 02:11:59.228196  4633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.003s
I20240503 02:11:59.231631  4647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:59.232993  4633 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 02:11:59.233143  4633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9c804fe67ee34e7fae5fab23db90f298"
format_stamp: "Formatted at 2024-05-03 02:11:59 on dist-test-slave-gq9h"
I20240503 02:11:59.233350  4633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:59.260524  4633 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:59.261446  4633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:59.261767  4633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:59.262832  4633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:59.264360  4633 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:59.264453  4633 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:59.264585  4633 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:59.264662  4633 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:59.294705  4633 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.6:36423
I20240503 02:11:59.294785  4759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.6:36423 every 8 connection(s)
I20240503 02:11:59.296085  4633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:11:59.304670   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 4633
I20240503 02:11:59.304836   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:11:59.308650   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.7:0
--local_ip_for_outbound_sockets=127.0.145.7
--webserver_interface=127.0.145.7
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--builtin_ntp_servers=127.0.145.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:59.310057  4760 heartbeater.cc:347] Connected to a master server at 127.0.145.62:40001
I20240503 02:11:59.310446  4760 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:59.311555  4760 heartbeater.cc:511] Master 127.0.145.62:40001 requested a full tablet report, sending...
I20240503 02:11:59.313203  3921 ts_manager.cc:194] Registered new tserver with Master: 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6:36423)
I20240503 02:11:59.313979  3921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.6:38035
I20240503 02:11:59.342352  4500 heartbeater.cc:503] Master 127.0.145.62:40001 was elected leader, sending a full tablet report...
W20240503 02:11:59.650000  4763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:59.650305  4763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:59.650439  4763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:59.658020  4763 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:59.658159  4763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:59.658331  4763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.7
I20240503 02:11:59.667614  4763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.145.7
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.7
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:59.668991  4763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:59.670787  4763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:59.683590  4769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:59.684193  4768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:59.685130  4771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:59.685910  4763 server_base.cc:1034] running on GCE node
I20240503 02:11:59.686590  4763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:59.687721  4763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:59.688983  4763 hybrid_clock.cc:648] HybridClock initialized: now 1714702319688898 us; error 63 us; skew 500 ppm
I20240503 02:11:59.689322  4763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:59.697341  4763 webserver.cc:460] Webserver started at http://127.0.145.7:36903/ using document root <none> and password file <none>
I20240503 02:11:59.698154  4763 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:59.698287  4763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:59.698652  4763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:59.703761  4763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "6c3559ccff5d4c6681b3c90f892143dc"
format_stamp: "Formatted at 2024-05-03 02:11:59 on dist-test-slave-gq9h"
I20240503 02:11:59.704610  4763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "6c3559ccff5d4c6681b3c90f892143dc"
format_stamp: "Formatted at 2024-05-03 02:11:59 on dist-test-slave-gq9h"
I20240503 02:11:59.710125  4763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240503 02:11:59.713757  4777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:59.715214  4763 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:11:59.715389  4763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6c3559ccff5d4c6681b3c90f892143dc"
format_stamp: "Formatted at 2024-05-03 02:11:59 on dist-test-slave-gq9h"
I20240503 02:11:59.715600  4763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:59.735287  4763 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:59.736193  4763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:59.736528  4763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:59.737646  4763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:59.739251  4763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:59.739356  4763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:59.739470  4763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:59.739542  4763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:59.771606  4763 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.7:35457
I20240503 02:11:59.771654  4889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.7:35457 every 8 connection(s)
I20240503 02:11:59.772936  4763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:11:59.774353   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 4763
I20240503 02:11:59.774646   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:11:59.786485  4890 heartbeater.cc:347] Connected to a master server at 127.0.145.62:40001
I20240503 02:11:59.786882  4890 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:59.787824  4890 heartbeater.cc:511] Master 127.0.145.62:40001 requested a full tablet report, sending...
I20240503 02:11:59.789637  3921 ts_manager.cc:194] Registered new tserver with Master: 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7:35457)
I20240503 02:11:59.790879  3921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.7:45493
I20240503 02:11:59.791290   580 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 02:11:59.810544  3921 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58766:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240503 02:11:59.825130  4630 heartbeater.cc:503] Master 127.0.145.62:40001 was elected leader, sending a full tablet report...
I20240503 02:11:59.850725  4174 tablet_service.cc:1467] Processing CreateTablet for tablet 1c1ff96ff85b4d5e9bd80fe5838b16db (DEFAULT_TABLE table=TestTable [id=a2f222c7915242918dc88c24b67a23ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:59.852380  4174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c1ff96ff85b4d5e9bd80fe5838b16db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:59.854842  4305 tablet_service.cc:1467] Processing CreateTablet for tablet 1c1ff96ff85b4d5e9bd80fe5838b16db (DEFAULT_TABLE table=TestTable [id=a2f222c7915242918dc88c24b67a23ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:59.856446  4305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c1ff96ff85b4d5e9bd80fe5838b16db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:59.866315  4906 tablet_bootstrap.cc:492] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2: Bootstrap starting.
I20240503 02:11:59.869227  4906 tablet_bootstrap.cc:654] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:59.869581  4907 tablet_bootstrap.cc:492] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc: Bootstrap starting.
I20240503 02:11:59.870417  4906 log.cc:824] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:59.872444  4907 tablet_bootstrap.cc:654] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:59.873106  4906 tablet_bootstrap.cc:492] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2: No bootstrap required, opened a new log
I20240503 02:11:59.873389  4906 ts_tablet_manager.cc:1367] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240503 02:11:59.874665  4907 log.cc:824] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:59.877405  4907 tablet_bootstrap.cc:492] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc: No bootstrap required, opened a new log
I20240503 02:11:59.877648  4907 ts_tablet_manager.cc:1367] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.003s
I20240503 02:11:59.880513  4906 raft_consensus.cc:348] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:11:59.880874  4906 raft_consensus.cc:374] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:59.880978  4906 raft_consensus.cc:729] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Initialized, Role: FOLLOWER
I20240503 02:11:59.881692  4906 consensus_queue.cc:260] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:11:59.883271  4906 ts_tablet_manager.cc:1398] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2: Time spent starting tablet: real 0.010s	user 0.011s	sys 0.000s
I20240503 02:11:59.884629  4907 raft_consensus.cc:348] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:11:59.884961  4907 raft_consensus.cc:374] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:59.885066  4907 raft_consensus.cc:729] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Initialized, Role: FOLLOWER
I20240503 02:11:59.885684  4907 consensus_queue.cc:260] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:11:59.887840  4907 ts_tablet_manager.cc:1398] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.004s
I20240503 02:11:59.995893  4911 raft_consensus.cc:482] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:59.996196  4911 raft_consensus.cc:504] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:11:59.997782  4911 leader_election.cc:288] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:00.004395  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c1ff96ff85b4d5e9bd80fe5838b16db" candidate_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f19d54f56849f495dc547de6a47ad2" is_pre_election: true
I20240503 02:12:00.004928  4194 raft_consensus.cc:2454] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b812251ceb44fbe8a54fea04a7f8dfc in term 0.
I20240503 02:12:00.005595  4260 leader_election.cc:302] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:00.006011  4911 raft_consensus.cc:2781] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:00.006163  4911 raft_consensus.cc:482] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:00.006263  4911 raft_consensus.cc:3037] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:00.008363  4911 raft_consensus.cc:504] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:00.008896  4911 leader_election.cc:288] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 election: Requested vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:00.009626  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c1ff96ff85b4d5e9bd80fe5838b16db" candidate_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
I20240503 02:12:00.009821  4194 raft_consensus.cc:3037] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:00.012144  4194 raft_consensus.cc:2454] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b812251ceb44fbe8a54fea04a7f8dfc in term 1.
I20240503 02:12:00.012576  4260 leader_election.cc:302] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:00.012889  4911 raft_consensus.cc:2781] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:00.013319  4911 raft_consensus.cc:686] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Becoming Leader. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Running, Role: LEADER
I20240503 02:12:00.013765  4911 consensus_queue.cc:237] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:00.018815  3921 catalog_manager.cc:5484] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3). New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:00.039062   580 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 02:12:00.041038   580 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c4af56cee8e546c384cf20f40deeacd7 to finish bootstrapping
I20240503 02:12:00.050465   580 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d7f19d54f56849f495dc547de6a47ad2 to finish bootstrapping
I20240503 02:12:00.057916   580 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8b812251ceb44fbe8a54fea04a7f8dfc to finish bootstrapping
I20240503 02:12:00.065280   580 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6ebeaff495e042c78067c5b18600dc98 to finish bootstrapping
I20240503 02:12:00.074147   580 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 37c60eb66a6b45aaac4e0d7061572da6 to finish bootstrapping
I20240503 02:12:00.082881   580 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9c804fe67ee34e7fae5fab23db90f298 to finish bootstrapping
I20240503 02:12:00.091542   580 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6c3559ccff5d4c6681b3c90f892143dc to finish bootstrapping
I20240503 02:12:00.100659   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 4373
I20240503 02:12:00.101625  4495 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:12:00.102444  4371 tablet.cc:2369] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:12:00.124138  4240 tablet.cc:2369] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:00.184216   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 4373
I20240503 02:12:00.200834   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 4503
I20240503 02:12:00.202013  4625 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:00.283223   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 4503
I20240503 02:12:00.299960   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 4633
I20240503 02:12:00.301054  4755 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:00.381222  4760 heartbeater.cc:503] Master 127.0.145.62:40001 was elected leader, sending a full tablet report...
I20240503 02:12:00.382205   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 4633
W20240503 02:12:00.398551  3908 connection.cc:535] server connection from 127.0.145.6:38035 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:00.398846   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 4763
I20240503 02:12:00.399948  4885 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:00.478804  4911 consensus_queue.cc:1035] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:00.481050   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 4763
I20240503 02:12:04.250507  3922 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58766:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:12:04.261607  4174 tablet_service.cc:1467] Processing CreateTablet for tablet 75a2da7ca68b43e28cad0847e4d27982 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3a36c423c3f74ce392ff73837341d077]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:04.261718  4305 tablet_service.cc:1467] Processing CreateTablet for tablet 8334f743021f4957a427800af656bec1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3a36c423c3f74ce392ff73837341d077]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:04.262076  4174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a2da7ca68b43e28cad0847e4d27982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:04.262390  4305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8334f743021f4957a427800af656bec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:04.266708  4940 tablet_bootstrap.cc:492] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2: Bootstrap starting.
I20240503 02:12:04.268429  4174 tablet_service.cc:1467] Processing CreateTablet for tablet d98afd14a7f14415a0f404dea8750d82 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3a36c423c3f74ce392ff73837341d077]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:04.268688  4940 tablet_bootstrap.cc:654] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:04.269035  4174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d98afd14a7f14415a0f404dea8750d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:04.274034  4942 tablet_bootstrap.cc:492] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc: Bootstrap starting.
I20240503 02:12:04.275331  4940 tablet_bootstrap.cc:492] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2: No bootstrap required, opened a new log
I20240503 02:12:04.275588  4940 ts_tablet_manager.cc:1367] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.004s
I20240503 02:12:04.276062  4942 tablet_bootstrap.cc:654] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:04.276885  4940 raft_consensus.cc:348] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:04.277273  4940 raft_consensus.cc:374] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:04.277428  4940 raft_consensus.cc:729] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Initialized, Role: FOLLOWER
I20240503 02:12:04.278084  4940 consensus_queue.cc:260] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:04.278827  4940 ts_tablet_manager.cc:1398] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 02:12:04.279064  4940 tablet_bootstrap.cc:492] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2: Bootstrap starting.
I20240503 02:12:04.279450  4942 tablet_bootstrap.cc:492] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc: No bootstrap required, opened a new log
I20240503 02:12:04.279686  4942 ts_tablet_manager.cc:1367] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240503 02:12:04.280889  4940 tablet_bootstrap.cc:654] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:04.281194  4942 raft_consensus.cc:348] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:04.281591  4942 raft_consensus.cc:374] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:04.281776  4942 raft_consensus.cc:729] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Initialized, Role: FOLLOWER
I20240503 02:12:04.282238  4942 consensus_queue.cc:260] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:04.289140  4940 tablet_bootstrap.cc:492] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2: No bootstrap required, opened a new log
I20240503 02:12:04.289379  4940 ts_tablet_manager.cc:1367] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.003s
I20240503 02:12:04.290513  4940 raft_consensus.cc:348] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:04.290875  4940 raft_consensus.cc:374] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:04.290988  4940 raft_consensus.cc:729] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Initialized, Role: FOLLOWER
I20240503 02:12:04.291334  4940 consensus_queue.cc:260] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:04.291045  4043 tablet_service.cc:1467] Processing CreateTablet for tablet 75a2da7ca68b43e28cad0847e4d27982 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3a36c423c3f74ce392ff73837341d077]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:04.291801  4940 ts_tablet_manager.cc:1398] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:04.292758  4043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a2da7ca68b43e28cad0847e4d27982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:04.292788  4041 tablet_service.cc:1467] Processing CreateTablet for tablet d98afd14a7f14415a0f404dea8750d82 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3a36c423c3f74ce392ff73837341d077]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:04.293282  4041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d98afd14a7f14415a0f404dea8750d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:04.295233  4042 tablet_service.cc:1467] Processing CreateTablet for tablet 8334f743021f4957a427800af656bec1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3a36c423c3f74ce392ff73837341d077]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:04.295773  4042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8334f743021f4957a427800af656bec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:04.299113  4942 ts_tablet_manager.cc:1398] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20240503 02:12:04.309592  4949 tablet_bootstrap.cc:492] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7: Bootstrap starting.
I20240503 02:12:04.312615  4949 tablet_bootstrap.cc:654] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:04.313819  4949 log.cc:824] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:04.316390  4949 tablet_bootstrap.cc:492] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7: No bootstrap required, opened a new log
I20240503 02:12:04.316650  4949 ts_tablet_manager.cc:1367] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240503 02:12:04.322770  4949 raft_consensus.cc:348] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:04.323210  4949 raft_consensus.cc:374] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:04.323323  4949 raft_consensus.cc:729] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Initialized, Role: FOLLOWER
I20240503 02:12:04.324141  4949 consensus_queue.cc:260] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:04.325942  4949 ts_tablet_manager.cc:1398] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
I20240503 02:12:04.326360  4949 tablet_bootstrap.cc:492] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7: Bootstrap starting.
I20240503 02:12:04.327646  4949 tablet_bootstrap.cc:654] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:04.329550  4949 tablet_bootstrap.cc:492] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7: No bootstrap required, opened a new log
I20240503 02:12:04.329677  4949 ts_tablet_manager.cc:1367] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:12:04.330540  4949 raft_consensus.cc:348] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:04.330753  4949 raft_consensus.cc:374] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:04.330847  4949 raft_consensus.cc:729] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Initialized, Role: FOLLOWER
I20240503 02:12:04.331053  4949 consensus_queue.cc:260] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:04.331471  4949 ts_tablet_manager.cc:1398] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:04.331744  4949 tablet_bootstrap.cc:492] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7: Bootstrap starting.
I20240503 02:12:04.333154  4949 tablet_bootstrap.cc:654] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:04.334584  4944 raft_consensus.cc:482] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:04.334791  4944 raft_consensus.cc:504] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:04.336126  4944 leader_election.cc:288] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:04.337002  4949 tablet_bootstrap.cc:492] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7: No bootstrap required, opened a new log
I20240503 02:12:04.337244  4949 ts_tablet_manager.cc:1367] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240503 02:12:04.338045  4949 raft_consensus.cc:348] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:04.338258  4949 raft_consensus.cc:374] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:04.338339  4949 raft_consensus.cc:729] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Initialized, Role: FOLLOWER
I20240503 02:12:04.338589  4949 consensus_queue.cc:260] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:04.339248  4949 ts_tablet_manager.cc:1398] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 02:12:04.343279  4063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a2da7ca68b43e28cad0847e4d27982" candidate_uuid: "d7f19d54f56849f495dc547de6a47ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4af56cee8e546c384cf20f40deeacd7" is_pre_election: true
I20240503 02:12:04.343828  4063 raft_consensus.cc:2454] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7f19d54f56849f495dc547de6a47ad2 in term 0.
I20240503 02:12:04.344408  4130 leader_election.cc:302] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c4af56cee8e546c384cf20f40deeacd7, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:04.344794  4944 raft_consensus.cc:2781] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:04.344914  4944 raft_consensus.cc:482] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:04.345005  4944 raft_consensus.cc:3037] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:04.347101  4944 raft_consensus.cc:504] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:04.347652  4944 leader_election.cc:288] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 election: Requested vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:04.348243  4063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a2da7ca68b43e28cad0847e4d27982" candidate_uuid: "d7f19d54f56849f495dc547de6a47ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4af56cee8e546c384cf20f40deeacd7"
I20240503 02:12:04.348434  4063 raft_consensus.cc:3037] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:04.350495  4063 raft_consensus.cc:2454] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f19d54f56849f495dc547de6a47ad2 in term 1.
I20240503 02:12:04.350968  4130 leader_election.cc:302] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c4af56cee8e546c384cf20f40deeacd7, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:04.351276  4944 raft_consensus.cc:2781] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:04.351727  4944 raft_consensus.cc:686] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Becoming Leader. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Running, Role: LEADER
I20240503 02:12:04.352141  4944 consensus_queue.cc:237] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:04.355525  3922 catalog_manager.cc:5484] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2). New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } }
W20240503 02:12:04.365505  4109 tablet.cc:2369] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:04.452821  4944 raft_consensus.cc:482] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:04.453063  4944 raft_consensus.cc:504] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:04.453554  4944 leader_election.cc:288] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:04.454213  4063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98afd14a7f14415a0f404dea8750d82" candidate_uuid: "d7f19d54f56849f495dc547de6a47ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4af56cee8e546c384cf20f40deeacd7" is_pre_election: true
I20240503 02:12:04.454473  4063 raft_consensus.cc:2454] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7f19d54f56849f495dc547de6a47ad2 in term 0.
I20240503 02:12:04.455009  4130 leader_election.cc:302] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c4af56cee8e546c384cf20f40deeacd7, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:04.455271  4944 raft_consensus.cc:2781] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:04.455353  4944 raft_consensus.cc:482] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:04.455433  4944 raft_consensus.cc:3037] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:04.457321  4944 raft_consensus.cc:504] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:04.457784  4944 leader_election.cc:288] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 election: Requested vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:04.458285  4063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98afd14a7f14415a0f404dea8750d82" candidate_uuid: "d7f19d54f56849f495dc547de6a47ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4af56cee8e546c384cf20f40deeacd7"
I20240503 02:12:04.458479  4063 raft_consensus.cc:3037] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:04.460469  4063 raft_consensus.cc:2454] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f19d54f56849f495dc547de6a47ad2 in term 1.
I20240503 02:12:04.460851  4130 leader_election.cc:302] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c4af56cee8e546c384cf20f40deeacd7, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:04.461112  4944 raft_consensus.cc:2781] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:04.461279  4944 raft_consensus.cc:686] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Becoming Leader. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Running, Role: LEADER
I20240503 02:12:04.461552  4944 consensus_queue.cc:237] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:04.464021  3922 catalog_manager.cc:5484] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2). New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:04.593339  4945 raft_consensus.cc:482] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:04.593555  4945 raft_consensus.cc:504] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:04.594164  4945 leader_election.cc:288] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:04.600078  4063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8334f743021f4957a427800af656bec1" candidate_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4af56cee8e546c384cf20f40deeacd7" is_pre_election: true
I20240503 02:12:04.600306  4063 raft_consensus.cc:2454] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b812251ceb44fbe8a54fea04a7f8dfc in term 0.
I20240503 02:12:04.600739  4261 leader_election.cc:302] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, c4af56cee8e546c384cf20f40deeacd7; no voters: 
I20240503 02:12:04.601042  4945 raft_consensus.cc:2781] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:04.601152  4945 raft_consensus.cc:482] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:04.601238  4945 raft_consensus.cc:3037] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:04.603192  4945 raft_consensus.cc:504] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:04.603684  4945 leader_election.cc:288] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 election: Requested vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:04.604403  4063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8334f743021f4957a427800af656bec1" candidate_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4af56cee8e546c384cf20f40deeacd7"
I20240503 02:12:04.604586  4063 raft_consensus.cc:3037] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:04.606475  4063 raft_consensus.cc:2454] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b812251ceb44fbe8a54fea04a7f8dfc in term 1.
I20240503 02:12:04.606943  4261 leader_election.cc:302] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, c4af56cee8e546c384cf20f40deeacd7; no voters: 
I20240503 02:12:04.607316  4945 raft_consensus.cc:2781] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:04.607484  4945 raft_consensus.cc:686] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Becoming Leader. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Running, Role: LEADER
I20240503 02:12:04.607769  4945 consensus_queue.cc:237] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:04.610898  3922 catalog_manager.cc:5484] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3). New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:04.852593  4944 consensus_queue.cc:1035] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:04.988637  4960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:04.989022  4960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:04.997238  4960 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:05.018776  4944 consensus_queue.cc:1035] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 02:12:05.159318  4063 raft_consensus.cc:1261] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:05.159970  4945 consensus_queue.cc:1035] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:05.194301  4998 mvcc.cc:205] Tried to move back new op lower bound from 7023420723844292608 to 7023420721595260928. Current Snapshot: MvccSnapshot[applied={T|T < 7023420723844292608}]

INSERT report
    rows total: 2000
    time total: 193.662 ms
  time per row: 0.0968312 ms
I20240503 02:12:05.350051  3919 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58766:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:12:05.359243  4043 tablet_service.cc:1467] Processing CreateTablet for tablet ccccc26d94514b87a91b0c00fa18e669 (DEFAULT_TABLE table=rebalance_test_table_1 [id=908c33982a334c0a957523734a0931d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:05.359200  4305 tablet_service.cc:1467] Processing CreateTablet for tablet ccccc26d94514b87a91b0c00fa18e669 (DEFAULT_TABLE table=rebalance_test_table_1 [id=908c33982a334c0a957523734a0931d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:05.359437  4304 tablet_service.cc:1467] Processing CreateTablet for tablet ed79dde0a1f4419c8085c042ed5b8780 (DEFAULT_TABLE table=rebalance_test_table_1 [id=908c33982a334c0a957523734a0931d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:05.359678  4043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccccc26d94514b87a91b0c00fa18e669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:05.359688  4305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccccc26d94514b87a91b0c00fa18e669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:05.359990  4304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed79dde0a1f4419c8085c042ed5b8780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:05.362263  4174 tablet_service.cc:1467] Processing CreateTablet for tablet ed79dde0a1f4419c8085c042ed5b8780 (DEFAULT_TABLE table=rebalance_test_table_1 [id=908c33982a334c0a957523734a0931d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:05.362720  4174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed79dde0a1f4419c8085c042ed5b8780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:05.362711  4303 tablet_service.cc:1467] Processing CreateTablet for tablet df156172a4554094b36cd5cc3876c922 (DEFAULT_TABLE table=rebalance_test_table_1 [id=908c33982a334c0a957523734a0931d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:05.363216  4303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df156172a4554094b36cd5cc3876c922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:05.363284  4041 tablet_service.cc:1467] Processing CreateTablet for tablet df156172a4554094b36cd5cc3876c922 (DEFAULT_TABLE table=rebalance_test_table_1 [id=908c33982a334c0a957523734a0931d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:05.363634  4041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df156172a4554094b36cd5cc3876c922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:05.376104  5018 tablet_bootstrap.cc:492] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc: Bootstrap starting.
I20240503 02:12:05.378226  5018 tablet_bootstrap.cc:654] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:05.379035  5019 tablet_bootstrap.cc:492] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2: Bootstrap starting.
I20240503 02:12:05.381006  5019 tablet_bootstrap.cc:654] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:05.387591  5018 tablet_bootstrap.cc:492] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc: No bootstrap required, opened a new log
I20240503 02:12:05.387821  5018 ts_tablet_manager.cc:1367] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent bootstrapping tablet: real 0.012s	user 0.010s	sys 0.000s
I20240503 02:12:05.388659  5018 raft_consensus.cc:348] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:05.388901  5018 raft_consensus.cc:374] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:05.389012  5018 raft_consensus.cc:729] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Initialized, Role: FOLLOWER
I20240503 02:12:05.389257  5018 consensus_queue.cc:260] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:05.391139  5018 ts_tablet_manager.cc:1398] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:12:05.391435  5018 tablet_bootstrap.cc:492] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc: Bootstrap starting.
I20240503 02:12:05.392231  5022 tablet_bootstrap.cc:492] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7: Bootstrap starting.
I20240503 02:12:05.395413  5018 tablet_bootstrap.cc:654] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:05.397117  5022 tablet_bootstrap.cc:654] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:05.399740  5019 tablet_bootstrap.cc:492] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2: No bootstrap required, opened a new log
I20240503 02:12:05.399811  5022 tablet_bootstrap.cc:492] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7: No bootstrap required, opened a new log
I20240503 02:12:05.399905  5019 ts_tablet_manager.cc:1367] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2: Time spent bootstrapping tablet: real 0.021s	user 0.011s	sys 0.008s
I20240503 02:12:05.399956  5022 ts_tablet_manager.cc:1367] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.000s
I20240503 02:12:05.400619  5022 raft_consensus.cc:348] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:05.400800  5022 raft_consensus.cc:374] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:05.400707  5019 raft_consensus.cc:348] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:05.400890  5022 raft_consensus.cc:729] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Initialized, Role: FOLLOWER
I20240503 02:12:05.400915  5019 raft_consensus.cc:374] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:05.400998  5019 raft_consensus.cc:729] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Initialized, Role: FOLLOWER
I20240503 02:12:05.401141  5022 consensus_queue.cc:260] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:05.401208  5019 consensus_queue.cc:260] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:05.401765  5019 ts_tablet_manager.cc:1398] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:05.401787  5022 ts_tablet_manager.cc:1398] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:05.402740  5022 tablet_bootstrap.cc:492] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7: Bootstrap starting.
I20240503 02:12:05.402740  5018 tablet_bootstrap.cc:492] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc: No bootstrap required, opened a new log
I20240503 02:12:05.402913  5018 ts_tablet_manager.cc:1367] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.000s
I20240503 02:12:05.403682  5018 raft_consensus.cc:348] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:05.403913  5018 raft_consensus.cc:374] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:05.403991  5018 raft_consensus.cc:729] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Initialized, Role: FOLLOWER
I20240503 02:12:05.404224  5018 consensus_queue.cc:260] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:05.404745  5018 ts_tablet_manager.cc:1398] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:05.405043  5018 tablet_bootstrap.cc:492] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc: Bootstrap starting.
I20240503 02:12:05.404675  5022 tablet_bootstrap.cc:654] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:05.406594  5018 tablet_bootstrap.cc:654] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:05.410375  5018 tablet_bootstrap.cc:492] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc: No bootstrap required, opened a new log
I20240503 02:12:05.410413  5022 tablet_bootstrap.cc:492] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7: No bootstrap required, opened a new log
I20240503 02:12:05.410516  5018 ts_tablet_manager.cc:1367] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240503 02:12:05.410622  5022 ts_tablet_manager.cc:1367] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7: Time spent bootstrapping tablet: real 0.009s	user 0.001s	sys 0.003s
I20240503 02:12:05.411216  5018 raft_consensus.cc:348] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:05.411260  5022 raft_consensus.cc:348] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:05.411376  5018 raft_consensus.cc:374] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:05.411448  5022 raft_consensus.cc:374] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:05.411446  5018 raft_consensus.cc:729] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Initialized, Role: FOLLOWER
I20240503 02:12:05.411520  5022 raft_consensus.cc:729] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Initialized, Role: FOLLOWER
I20240503 02:12:05.411680  5018 consensus_queue.cc:260] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:05.411772  5022 consensus_queue.cc:260] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:05.412159  5018 ts_tablet_manager.cc:1398] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240503 02:12:05.413448  5022 ts_tablet_manager.cc:1398] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 02:12:05.475132  4961 raft_consensus.cc:482] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:05.475384  4961 raft_consensus.cc:504] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:05.476828  4961 leader_election.cc:288] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:05.482760  5012 raft_consensus.cc:482] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:05.482973  5012 raft_consensus.cc:504] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:05.483104  4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccccc26d94514b87a91b0c00fa18e669" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" is_pre_election: true
I20240503 02:12:05.483549  4325 raft_consensus.cc:2454] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 0.
I20240503 02:12:05.483585  5012 leader_election.cc:288] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:05.484087  4061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df156172a4554094b36cd5cc3876c922" candidate_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4af56cee8e546c384cf20f40deeacd7" is_pre_election: true
I20240503 02:12:05.484138  3997 leader_election.cc:302] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, c4af56cee8e546c384cf20f40deeacd7; no voters: 
I20240503 02:12:05.484328  4061 raft_consensus.cc:2454] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b812251ceb44fbe8a54fea04a7f8dfc in term 0.
I20240503 02:12:05.484545  4961 raft_consensus.cc:2781] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:05.484681  4961 raft_consensus.cc:482] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:05.484755  4961 raft_consensus.cc:3037] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:05.484750  4261 leader_election.cc:302] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, c4af56cee8e546c384cf20f40deeacd7; no voters: 
I20240503 02:12:05.484990  5012 raft_consensus.cc:2781] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:05.485092  5012 raft_consensus.cc:482] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:05.485168  5012 raft_consensus.cc:3037] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:05.487670  4961 raft_consensus.cc:504] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:05.487864  5012 raft_consensus.cc:504] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:05.488276  4961 leader_election.cc:288] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 election: Requested vote from peers 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:05.488312  5012 leader_election.cc:288] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 election: Requested vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:05.488801  4061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df156172a4554094b36cd5cc3876c922" candidate_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4af56cee8e546c384cf20f40deeacd7"
I20240503 02:12:05.488832  4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccccc26d94514b87a91b0c00fa18e669" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc"
I20240503 02:12:05.488963  4325 raft_consensus.cc:3037] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:05.488983  4061 raft_consensus.cc:3037] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:05.490849  4061 raft_consensus.cc:2454] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b812251ceb44fbe8a54fea04a7f8dfc in term 1.
I20240503 02:12:05.490962  4325 raft_consensus.cc:2454] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 1.
I20240503 02:12:05.491281  4261 leader_election.cc:302] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, c4af56cee8e546c384cf20f40deeacd7; no voters: 
I20240503 02:12:05.491374  3997 leader_election.cc:302] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, c4af56cee8e546c384cf20f40deeacd7; no voters: 
I20240503 02:12:05.491623  5012 raft_consensus.cc:2781] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:05.491703  4961 raft_consensus.cc:2781] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:05.491904  5012 raft_consensus.cc:686] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Becoming Leader. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Running, Role: LEADER
I20240503 02:12:05.492168  4961 raft_consensus.cc:686] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Becoming Leader. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Running, Role: LEADER
I20240503 02:12:05.492230  5012 consensus_queue.cc:237] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:05.492693  4961 consensus_queue.cc:237] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:05.495710  3922 catalog_manager.cc:5484] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3). New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:05.496131  3919 catalog_manager.cc:5484] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:05.743171  4944 raft_consensus.cc:482] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:05.743353  4944 raft_consensus.cc:504] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:05.743944  4944 leader_election.cc:288] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:05.749181  4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed79dde0a1f4419c8085c042ed5b8780" candidate_uuid: "d7f19d54f56849f495dc547de6a47ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" is_pre_election: true
I20240503 02:12:05.749436  4325 raft_consensus.cc:2454] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7f19d54f56849f495dc547de6a47ad2 in term 0.
I20240503 02:12:05.749877  4128 leader_election.cc:302] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:05.750177  4944 raft_consensus.cc:2781] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:05.750299  4944 raft_consensus.cc:482] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:05.750399  4944 raft_consensus.cc:3037] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:05.752588  4944 raft_consensus.cc:504] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:05.753065  4944 leader_election.cc:288] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 election: Requested vote from peers 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:05.753594  4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed79dde0a1f4419c8085c042ed5b8780" candidate_uuid: "d7f19d54f56849f495dc547de6a47ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc"
I20240503 02:12:05.753795  4325 raft_consensus.cc:3037] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:05.755724  4325 raft_consensus.cc:2454] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f19d54f56849f495dc547de6a47ad2 in term 1.
I20240503 02:12:05.756170  4128 leader_election.cc:302] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:05.756496  4944 raft_consensus.cc:2781] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:05.756641  4944 raft_consensus.cc:686] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Becoming Leader. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Running, Role: LEADER
I20240503 02:12:05.756904  4944 consensus_queue.cc:237] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:05.759294  3919 catalog_manager.cc:5484] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2). New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:05.926239  5012 consensus_queue.cc:1035] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:05.952993  4961 consensus_queue.cc:1035] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:06.185269  5035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:06.185681  5035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:06.193189  5035 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:12:06.367426  4944 consensus_queue.cc:1035] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 568.693 ms
  time per row: 0.284347 ms
I20240503 02:12:06.922585  3920 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58766:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:12:06.932204  4041 tablet_service.cc:1467] Processing CreateTablet for tablet 368fca99d20d4d6e826c45e44379c056 (DEFAULT_TABLE table=rebalance_test_table_2 [id=46a3ef9b51d44a84a25ae9ff7198c9b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:06.932430  4174 tablet_service.cc:1467] Processing CreateTablet for tablet 368fca99d20d4d6e826c45e44379c056 (DEFAULT_TABLE table=rebalance_test_table_2 [id=46a3ef9b51d44a84a25ae9ff7198c9b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:06.932677  4041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368fca99d20d4d6e826c45e44379c056. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:06.932876  4174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368fca99d20d4d6e826c45e44379c056. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:06.933301  4043 tablet_service.cc:1467] Processing CreateTablet for tablet d846ad6206514c79b142064043ad1cb0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=46a3ef9b51d44a84a25ae9ff7198c9b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:06.933681  4043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d846ad6206514c79b142064043ad1cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:06.936544  4173 tablet_service.cc:1467] Processing CreateTablet for tablet ec5c9e506c2f4c29aea9a9de259a0c19 (DEFAULT_TABLE table=rebalance_test_table_2 [id=46a3ef9b51d44a84a25ae9ff7198c9b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:06.936640  4172 tablet_service.cc:1467] Processing CreateTablet for tablet d846ad6206514c79b142064043ad1cb0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=46a3ef9b51d44a84a25ae9ff7198c9b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:06.937742  4173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec5c9e506c2f4c29aea9a9de259a0c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:06.937810  4305 tablet_service.cc:1467] Processing CreateTablet for tablet ec5c9e506c2f4c29aea9a9de259a0c19 (DEFAULT_TABLE table=rebalance_test_table_2 [id=46a3ef9b51d44a84a25ae9ff7198c9b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:06.938212  4305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec5c9e506c2f4c29aea9a9de259a0c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:06.942858  4172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d846ad6206514c79b142064043ad1cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:06.946202  5089 tablet_bootstrap.cc:492] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7: Bootstrap starting.
I20240503 02:12:06.948436  5089 tablet_bootstrap.cc:654] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:06.951368  5090 tablet_bootstrap.cc:492] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2: Bootstrap starting.
I20240503 02:12:06.953086  5089 tablet_bootstrap.cc:492] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7: No bootstrap required, opened a new log
I20240503 02:12:06.953220  5090 tablet_bootstrap.cc:654] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:06.953235  5089 ts_tablet_manager.cc:1367] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240503 02:12:06.953886  5092 tablet_bootstrap.cc:492] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc: Bootstrap starting.
I20240503 02:12:06.954030  5089 raft_consensus.cc:348] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:06.954319  5089 raft_consensus.cc:374] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:06.954490  5089 raft_consensus.cc:729] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Initialized, Role: FOLLOWER
I20240503 02:12:06.954856  5089 consensus_queue.cc:260] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:06.955168  5090 tablet_bootstrap.cc:492] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2: No bootstrap required, opened a new log
I20240503 02:12:06.955324  5090 ts_tablet_manager.cc:1367] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2: Time spent bootstrapping tablet: real 0.004s	user 0.004s	sys 0.000s
I20240503 02:12:06.955411  5089 ts_tablet_manager.cc:1398] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:06.955703  5089 tablet_bootstrap.cc:492] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7: Bootstrap starting.
I20240503 02:12:06.956072  5090 raft_consensus.cc:348] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:06.956236  5092 tablet_bootstrap.cc:654] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:06.956284  5090 raft_consensus.cc:374] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:06.956400  5090 raft_consensus.cc:729] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Initialized, Role: FOLLOWER
I20240503 02:12:06.956647  5090 consensus_queue.cc:260] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:06.957231  5090 ts_tablet_manager.cc:1398] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:12:06.957343  5089 tablet_bootstrap.cc:654] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:06.957547  5090 tablet_bootstrap.cc:492] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2: Bootstrap starting.
I20240503 02:12:06.959156  5090 tablet_bootstrap.cc:654] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:06.963675  5089 tablet_bootstrap.cc:492] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7: No bootstrap required, opened a new log
I20240503 02:12:06.963991  5089 ts_tablet_manager.cc:1367] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20240503 02:12:06.965092  5089 raft_consensus.cc:348] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:06.965479  5089 raft_consensus.cc:374] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:06.965618  5089 raft_consensus.cc:729] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Initialized, Role: FOLLOWER
I20240503 02:12:06.966028  5089 consensus_queue.cc:260] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:06.966856  5089 ts_tablet_manager.cc:1398] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:12:06.967649  5090 tablet_bootstrap.cc:492] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2: No bootstrap required, opened a new log
I20240503 02:12:06.967810  5090 ts_tablet_manager.cc:1367] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.007s
I20240503 02:12:06.968422  5090 raft_consensus.cc:348] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:06.968591  5090 raft_consensus.cc:374] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:06.968667  5090 raft_consensus.cc:729] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Initialized, Role: FOLLOWER
I20240503 02:12:06.968887  5090 consensus_queue.cc:260] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:06.969347  5090 ts_tablet_manager.cc:1398] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.002s
I20240503 02:12:06.969599  5090 tablet_bootstrap.cc:492] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2: Bootstrap starting.
I20240503 02:12:06.970957  5090 tablet_bootstrap.cc:654] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:06.971813  5092 tablet_bootstrap.cc:492] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc: No bootstrap required, opened a new log
I20240503 02:12:06.972055  5092 ts_tablet_manager.cc:1367] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent bootstrapping tablet: real 0.018s	user 0.012s	sys 0.000s
I20240503 02:12:06.972910  5092 raft_consensus.cc:348] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:06.973204  5092 raft_consensus.cc:374] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:06.973352  5092 raft_consensus.cc:729] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Initialized, Role: FOLLOWER
I20240503 02:12:06.973670  5092 consensus_queue.cc:260] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:06.973827  5090 tablet_bootstrap.cc:492] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2: No bootstrap required, opened a new log
I20240503 02:12:06.973956  5090 ts_tablet_manager.cc:1367] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:12:06.974356  5092 ts_tablet_manager.cc:1398] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:06.974964  5090 raft_consensus.cc:348] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:06.975162  5090 raft_consensus.cc:374] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:06.975267  5090 raft_consensus.cc:729] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Initialized, Role: FOLLOWER
I20240503 02:12:06.975517  5090 consensus_queue.cc:260] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:06.976050  5090 ts_tablet_manager.cc:1398] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:12:07.032776  4961 raft_consensus.cc:482] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:07.032967  4961 raft_consensus.cc:504] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:07.033553  4961 leader_election.cc:288] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:07.036190  5002 raft_consensus.cc:482] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:07.036377  5002 raft_consensus.cc:504] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:07.037039  5002 leader_election.cc:288] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:07.037624  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec5c9e506c2f4c29aea9a9de259a0c19" candidate_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f19d54f56849f495dc547de6a47ad2" is_pre_election: true
I20240503 02:12:07.037859  4194 raft_consensus.cc:2454] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b812251ceb44fbe8a54fea04a7f8dfc in term 0.
I20240503 02:12:07.038286  4260 leader_election.cc:302] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:07.038547  5002 raft_consensus.cc:2781] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:07.038687  5002 raft_consensus.cc:482] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:07.038775  5002 raft_consensus.cc:3037] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:07.040452  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d846ad6206514c79b142064043ad1cb0" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f19d54f56849f495dc547de6a47ad2" is_pre_election: true
I20240503 02:12:07.040696  4194 raft_consensus.cc:2454] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 0.
I20240503 02:12:07.041015  5002 raft_consensus.cc:504] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:07.041129  3998 leader_election.cc:302] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c4af56cee8e546c384cf20f40deeacd7, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:07.041453  4961 raft_consensus.cc:2781] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:07.041515  5002 leader_election.cc:288] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 election: Requested vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:07.041586  4961 raft_consensus.cc:482] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:07.041697  4961 raft_consensus.cc:3037] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:07.041936  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec5c9e506c2f4c29aea9a9de259a0c19" candidate_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
I20240503 02:12:07.042165  4194 raft_consensus.cc:3037] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:07.043678  4961 raft_consensus.cc:504] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:07.044026  4194 raft_consensus.cc:2454] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b812251ceb44fbe8a54fea04a7f8dfc in term 1.
I20240503 02:12:07.044138  4961 leader_election.cc:288] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 election: Requested vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:07.044466  4260 leader_election.cc:302] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:07.044672  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d846ad6206514c79b142064043ad1cb0" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
I20240503 02:12:07.044777  5002 raft_consensus.cc:2781] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:07.044878  4194 raft_consensus.cc:3037] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:07.045169  5002 raft_consensus.cc:686] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Becoming Leader. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Running, Role: LEADER
I20240503 02:12:07.045513  5002 consensus_queue.cc:237] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:07.047245  4194 raft_consensus.cc:2454] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 1.
I20240503 02:12:07.047588  3998 leader_election.cc:302] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c4af56cee8e546c384cf20f40deeacd7, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:07.047869  4961 raft_consensus.cc:2781] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:07.048259  4961 raft_consensus.cc:686] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Becoming Leader. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Running, Role: LEADER
I20240503 02:12:07.048578  4961 consensus_queue.cc:237] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:07.048480  3920 catalog_manager.cc:5484] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3). New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:07.051079  3914 catalog_manager.cc:5484] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:07.180934  4961 raft_consensus.cc:482] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:07.181154  4961 raft_consensus.cc:504] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:07.181658  4961 leader_election.cc:288] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:07.182188  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368fca99d20d4d6e826c45e44379c056" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f19d54f56849f495dc547de6a47ad2" is_pre_election: true
I20240503 02:12:07.182437  4194 raft_consensus.cc:2454] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 0.
I20240503 02:12:07.182863  3998 leader_election.cc:302] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c4af56cee8e546c384cf20f40deeacd7, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:07.183157  4961 raft_consensus.cc:2781] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:07.183247  4961 raft_consensus.cc:482] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:07.183351  4961 raft_consensus.cc:3037] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:07.185199  4961 raft_consensus.cc:504] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:07.185669  4961 leader_election.cc:288] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 election: Requested vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:07.186156  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368fca99d20d4d6e826c45e44379c056" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
I20240503 02:12:07.186352  4194 raft_consensus.cc:3037] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:07.188249  4194 raft_consensus.cc:2454] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 1.
I20240503 02:12:07.188674  3998 leader_election.cc:302] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c4af56cee8e546c384cf20f40deeacd7, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:07.188949  4961 raft_consensus.cc:2781] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:07.189107  4961 raft_consensus.cc:686] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Becoming Leader. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Running, Role: LEADER
I20240503 02:12:07.189342  4961 consensus_queue.cc:237] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:07.191783  3914 catalog_manager.cc:5484] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:07.476338  5002 consensus_queue.cc:1035] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:07.580149  4961 consensus_queue.cc:1035] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:07.590463  5105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:07.590869  5105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:07.598383  5105 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 02:12:07.648440  4961 consensus_queue.cc:1035] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 187.005 ms
  time per row: 0.0935025 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240503 02:12:08.866317  3915 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58766:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:12:08.875612  4305 tablet_service.cc:1467] Processing CreateTablet for tablet 87e472f1975b4141b795aa6c3b75c7c0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76e610e6ec844c2da65036194e355bbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:08.876080  4305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e472f1975b4141b795aa6c3b75c7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:08.876611  4303 tablet_service.cc:1467] Processing CreateTablet for tablet 07df29aeb41b427a8dcaf4a1a7db30da (DEFAULT_TABLE table=rebalance_test_table_3 [id=76e610e6ec844c2da65036194e355bbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:08.876710  4304 tablet_service.cc:1467] Processing CreateTablet for tablet 8b76c95bac0f497f8ab21c5295028026 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76e610e6ec844c2da65036194e355bbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:08.876833  4174 tablet_service.cc:1467] Processing CreateTablet for tablet 07df29aeb41b427a8dcaf4a1a7db30da (DEFAULT_TABLE table=rebalance_test_table_3 [id=76e610e6ec844c2da65036194e355bbe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:08.877036  4303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07df29aeb41b427a8dcaf4a1a7db30da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:08.877224  4174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07df29aeb41b427a8dcaf4a1a7db30da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:08.877790  4304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b76c95bac0f497f8ab21c5295028026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:08.883116  4043 tablet_service.cc:1467] Processing CreateTablet for tablet 87e472f1975b4141b795aa6c3b75c7c0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76e610e6ec844c2da65036194e355bbe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:08.883672  4043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e472f1975b4141b795aa6c3b75c7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:08.884389  4041 tablet_service.cc:1467] Processing CreateTablet for tablet 8b76c95bac0f497f8ab21c5295028026 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76e610e6ec844c2da65036194e355bbe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:08.884763  4041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b76c95bac0f497f8ab21c5295028026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:08.890255  5161 tablet_bootstrap.cc:492] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7: Bootstrap starting.
I20240503 02:12:08.892997  5161 tablet_bootstrap.cc:654] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:08.893052  5162 tablet_bootstrap.cc:492] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc: Bootstrap starting.
I20240503 02:12:08.893895  5160 tablet_bootstrap.cc:492] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2: Bootstrap starting.
I20240503 02:12:08.894845  5162 tablet_bootstrap.cc:654] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:08.896152  5160 tablet_bootstrap.cc:654] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:08.898711  5160 tablet_bootstrap.cc:492] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2: No bootstrap required, opened a new log
I20240503 02:12:08.898742  5161 tablet_bootstrap.cc:492] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7: No bootstrap required, opened a new log
I20240503 02:12:08.898866  5160 ts_tablet_manager.cc:1367] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:12:08.898888  5161 ts_tablet_manager.cc:1367] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7: Time spent bootstrapping tablet: real 0.009s	user 0.008s	sys 0.000s
I20240503 02:12:08.899268  5162 tablet_bootstrap.cc:492] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc: No bootstrap required, opened a new log
I20240503 02:12:08.899421  5162 ts_tablet_manager.cc:1367] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240503 02:12:08.899523  5160 raft_consensus.cc:348] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:08.899698  5160 raft_consensus.cc:374] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:08.899673  5161 raft_consensus.cc:348] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:08.899794  5160 raft_consensus.cc:729] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Initialized, Role: FOLLOWER
I20240503 02:12:08.899881  5161 raft_consensus.cc:374] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:08.900014  5161 raft_consensus.cc:729] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Initialized, Role: FOLLOWER
I20240503 02:12:08.900031  5160 consensus_queue.cc:260] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:08.900189  5162 raft_consensus.cc:348] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:08.900259  5161 consensus_queue.cc:260] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:08.900415  5162 raft_consensus.cc:374] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:08.900599  5160 ts_tablet_manager.cc:1398] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:08.900527  5162 raft_consensus.cc:729] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Initialized, Role: FOLLOWER
I20240503 02:12:08.901348  5162 consensus_queue.cc:260] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:08.902856  5161 ts_tablet_manager.cc:1398] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:12:08.903194  5161 tablet_bootstrap.cc:492] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7: Bootstrap starting.
I20240503 02:12:08.905102  5161 tablet_bootstrap.cc:654] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:08.905792  5162 ts_tablet_manager.cc:1398] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240503 02:12:08.906219  5162 tablet_bootstrap.cc:492] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc: Bootstrap starting.
I20240503 02:12:08.907986  5162 tablet_bootstrap.cc:654] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:08.908429  5161 tablet_bootstrap.cc:492] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7: No bootstrap required, opened a new log
I20240503 02:12:08.908599  5161 ts_tablet_manager.cc:1367] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:12:08.909344  5161 raft_consensus.cc:348] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:08.909579  5161 raft_consensus.cc:374] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:08.909669  5161 raft_consensus.cc:729] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Initialized, Role: FOLLOWER
I20240503 02:12:08.909886  5161 consensus_queue.cc:260] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:08.909992  5162 tablet_bootstrap.cc:492] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc: No bootstrap required, opened a new log
I20240503 02:12:08.910135  5162 ts_tablet_manager.cc:1367] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:12:08.910370  5161 ts_tablet_manager.cc:1398] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:08.911005  5162 raft_consensus.cc:348] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:08.911207  5162 raft_consensus.cc:374] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:08.911288  5162 raft_consensus.cc:729] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Initialized, Role: FOLLOWER
I20240503 02:12:08.911509  5162 consensus_queue.cc:260] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:08.911921  5162 ts_tablet_manager.cc:1398] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:08.912235  5162 tablet_bootstrap.cc:492] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc: Bootstrap starting.
I20240503 02:12:08.913812  5162 tablet_bootstrap.cc:654] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:08.915712  5162 tablet_bootstrap.cc:492] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc: No bootstrap required, opened a new log
I20240503 02:12:08.915910  5162 ts_tablet_manager.cc:1367] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:12:08.916501  5162 raft_consensus.cc:348] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:08.916649  5162 raft_consensus.cc:374] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:08.916714  5162 raft_consensus.cc:729] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Initialized, Role: FOLLOWER
I20240503 02:12:08.916886  5162 consensus_queue.cc:260] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:08.917289  5162 ts_tablet_manager.cc:1398] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:12:08.917335  5079 raft_consensus.cc:482] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:08.917506  5079 raft_consensus.cc:504] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:08.918116  5079 leader_election.cc:288] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:08.918504  4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07df29aeb41b427a8dcaf4a1a7db30da" candidate_uuid: "d7f19d54f56849f495dc547de6a47ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" is_pre_election: true
I20240503 02:12:08.918700  4325 raft_consensus.cc:2454] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7f19d54f56849f495dc547de6a47ad2 in term 0.
I20240503 02:12:08.919102  4128 leader_election.cc:302] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:08.919360  5079 raft_consensus.cc:2781] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:08.919499  5079 raft_consensus.cc:482] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:08.919586  5079 raft_consensus.cc:3037] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:08.921846  5079 raft_consensus.cc:504] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:08.922312  5079 leader_election.cc:288] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 election: Requested vote from peers 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:08.922796  4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07df29aeb41b427a8dcaf4a1a7db30da" candidate_uuid: "d7f19d54f56849f495dc547de6a47ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc"
I20240503 02:12:08.922996  4325 raft_consensus.cc:3037] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:08.925140  4325 raft_consensus.cc:2454] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f19d54f56849f495dc547de6a47ad2 in term 1.
I20240503 02:12:08.925522  4128 leader_election.cc:302] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:08.925817  5079 raft_consensus.cc:2781] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:08.926146  5079 raft_consensus.cc:686] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Becoming Leader. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Running, Role: LEADER
I20240503 02:12:08.926393  5079 consensus_queue.cc:237] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:08.928710  3918 catalog_manager.cc:5484] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2). New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:09.281282  4961 raft_consensus.cc:482] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:09.281478  4961 raft_consensus.cc:504] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:09.282140  4961 leader_election.cc:288] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:09.282879  4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b76c95bac0f497f8ab21c5295028026" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" is_pre_election: true
I20240503 02:12:09.283093  4325 raft_consensus.cc:2454] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 0.
I20240503 02:12:09.283495  3997 leader_election.cc:302] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, c4af56cee8e546c384cf20f40deeacd7; no voters: 
I20240503 02:12:09.283744  4961 raft_consensus.cc:2781] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:09.283890  4961 raft_consensus.cc:482] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:09.283991  4961 raft_consensus.cc:3037] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:09.286190  4961 raft_consensus.cc:504] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:09.286681  4961 leader_election.cc:288] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 election: Requested vote from peers 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:09.287185  4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b76c95bac0f497f8ab21c5295028026" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc"
I20240503 02:12:09.287369  4325 raft_consensus.cc:3037] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:09.289307  4325 raft_consensus.cc:2454] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 1.
I20240503 02:12:09.289762  3997 leader_election.cc:302] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, c4af56cee8e546c384cf20f40deeacd7; no voters: 
I20240503 02:12:09.290050  4961 raft_consensus.cc:2781] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:09.290409  4961 raft_consensus.cc:686] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Becoming Leader. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Running, Role: LEADER
I20240503 02:12:09.290745  4961 consensus_queue.cc:237] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:09.293463  3918 catalog_manager.cc:5484] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:09.331151  4961 raft_consensus.cc:482] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:09.331431  4961 raft_consensus.cc:504] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:09.331971  4961 leader_election.cc:288] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:09.332674  4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e472f1975b4141b795aa6c3b75c7c0" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" is_pre_election: true
I20240503 02:12:09.332966  4325 raft_consensus.cc:2454] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 0.
I20240503 02:12:09.333421  3997 leader_election.cc:302] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, c4af56cee8e546c384cf20f40deeacd7; no voters: 
I20240503 02:12:09.333724  4961 raft_consensus.cc:2781] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:09.333814  4961 raft_consensus.cc:482] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:09.333925  4961 raft_consensus.cc:3037] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:09.335896  4961 raft_consensus.cc:504] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:09.336426  4961 leader_election.cc:288] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 election: Requested vote from peers 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:09.336990  4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e472f1975b4141b795aa6c3b75c7c0" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc"
I20240503 02:12:09.337204  4325 raft_consensus.cc:3037] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:09.339390  4325 raft_consensus.cc:2454] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 1.
I20240503 02:12:09.339776  3997 leader_election.cc:302] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, c4af56cee8e546c384cf20f40deeacd7; no voters: 
I20240503 02:12:09.340058  4961 raft_consensus.cc:2781] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:09.340224  4961 raft_consensus.cc:686] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Becoming Leader. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Running, Role: LEADER
I20240503 02:12:09.340454  4961 consensus_queue.cc:237] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } }
I20240503 02:12:09.343225  3918 catalog_manager.cc:5484] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:09.491360  5079 consensus_queue.cc:1035] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:09.698508  4961 consensus_queue.cc:1035] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:09.815063  5174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:09.815403  5174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:09.823103  5174 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:09.829640  5171 consensus_queue.cc:1035] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'

INSERT report
    rows total: 2000
    time total: 145.844 ms
  time per row: 0.0729219 ms
I20240503 02:12:10.123314  3915 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58766:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:12:10.132706  4043 tablet_service.cc:1467] Processing CreateTablet for tablet 6e9898815c904092af0a6f238ed961bd (DEFAULT_TABLE table=rebalance_test_table_4 [id=7bd39e7281604b6ebd6334a68999befc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:10.132961  4174 tablet_service.cc:1467] Processing CreateTablet for tablet 6e9898815c904092af0a6f238ed961bd (DEFAULT_TABLE table=rebalance_test_table_4 [id=7bd39e7281604b6ebd6334a68999befc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:10.133168  4043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9898815c904092af0a6f238ed961bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:10.133425  4174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9898815c904092af0a6f238ed961bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:10.134042  4172 tablet_service.cc:1467] Processing CreateTablet for tablet 27609e0e32a34efc915bbff6750f8234 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7bd39e7281604b6ebd6334a68999befc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:10.134141  4305 tablet_service.cc:1467] Processing CreateTablet for tablet 27609e0e32a34efc915bbff6750f8234 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7bd39e7281604b6ebd6334a68999befc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:10.134442  4172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27609e0e32a34efc915bbff6750f8234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:10.134577  4305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27609e0e32a34efc915bbff6750f8234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:10.137174  4173 tablet_service.cc:1467] Processing CreateTablet for tablet 15fafdc84fec4b46979c311f377fd312 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7bd39e7281604b6ebd6334a68999befc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:10.137579  4173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15fafdc84fec4b46979c311f377fd312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:10.143318  4041 tablet_service.cc:1467] Processing CreateTablet for tablet 15fafdc84fec4b46979c311f377fd312 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7bd39e7281604b6ebd6334a68999befc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:10.144351  4041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15fafdc84fec4b46979c311f377fd312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:10.147166  5226 tablet_bootstrap.cc:492] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2: Bootstrap starting.
I20240503 02:12:10.149036  5226 tablet_bootstrap.cc:654] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:10.154116  5228 tablet_bootstrap.cc:492] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7: Bootstrap starting.
I20240503 02:12:10.154263  5227 tablet_bootstrap.cc:492] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc: Bootstrap starting.
I20240503 02:12:10.156139  5228 tablet_bootstrap.cc:654] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:10.156178  5226 tablet_bootstrap.cc:492] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2: No bootstrap required, opened a new log
I20240503 02:12:10.156258  5227 tablet_bootstrap.cc:654] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:10.156329  5226 ts_tablet_manager.cc:1367] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.006s
I20240503 02:12:10.157238  5226 raft_consensus.cc:348] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:10.157653  5226 raft_consensus.cc:374] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:10.157830  5226 raft_consensus.cc:729] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Initialized, Role: FOLLOWER
I20240503 02:12:10.158202  5226 consensus_queue.cc:260] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:10.159597  5226 ts_tablet_manager.cc:1398] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:12:10.159942  5226 tablet_bootstrap.cc:492] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2: Bootstrap starting.
I20240503 02:12:10.159987  5228 tablet_bootstrap.cc:492] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7: No bootstrap required, opened a new log
I20240503 02:12:10.160099  5227 tablet_bootstrap.cc:492] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc: No bootstrap required, opened a new log
I20240503 02:12:10.160140  5228 ts_tablet_manager.cc:1367] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.006s
I20240503 02:12:10.160228  5227 ts_tablet_manager.cc:1367] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.003s
I20240503 02:12:10.160923  5228 raft_consensus.cc:348] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:10.160923  5227 raft_consensus.cc:348] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:10.161137  5228 raft_consensus.cc:374] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:10.161214  5227 raft_consensus.cc:374] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:10.161226  5228 raft_consensus.cc:729] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Initialized, Role: FOLLOWER
I20240503 02:12:10.161304  5227 raft_consensus.cc:729] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Initialized, Role: FOLLOWER
I20240503 02:12:10.161450  5228 consensus_queue.cc:260] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:10.161549  5227 consensus_queue.cc:260] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:10.161675  5226 tablet_bootstrap.cc:654] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:10.162057  5227 ts_tablet_manager.cc:1398] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:12:10.162724  5228 ts_tablet_manager.cc:1398] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:12:10.163048  5228 tablet_bootstrap.cc:492] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7: Bootstrap starting.
I20240503 02:12:10.164876  5228 tablet_bootstrap.cc:654] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:10.169284  5228 tablet_bootstrap.cc:492] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7: No bootstrap required, opened a new log
I20240503 02:12:10.169456  5228 ts_tablet_manager.cc:1367] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240503 02:12:10.169814  5226 tablet_bootstrap.cc:492] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2: No bootstrap required, opened a new log
I20240503 02:12:10.169981  5226 ts_tablet_manager.cc:1367] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.004s
I20240503 02:12:10.170305  5228 raft_consensus.cc:348] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:10.170533  5228 raft_consensus.cc:374] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:10.170694  5228 raft_consensus.cc:729] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Initialized, Role: FOLLOWER
I20240503 02:12:10.170629  5226 raft_consensus.cc:348] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:10.170836  5226 raft_consensus.cc:374] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:10.170925  5226 raft_consensus.cc:729] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Initialized, Role: FOLLOWER
I20240503 02:12:10.170992  5228 consensus_queue.cc:260] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:10.171168  5226 consensus_queue.cc:260] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:10.171581  5228 ts_tablet_manager.cc:1398] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.002s
I20240503 02:12:10.171754  5226 ts_tablet_manager.cc:1398] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:12:10.172060  5226 tablet_bootstrap.cc:492] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2: Bootstrap starting.
I20240503 02:12:10.173756  5226 tablet_bootstrap.cc:654] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:10.176533  5226 tablet_bootstrap.cc:492] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2: No bootstrap required, opened a new log
I20240503 02:12:10.176684  5226 ts_tablet_manager.cc:1367] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.002s
I20240503 02:12:10.177297  5226 raft_consensus.cc:348] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:10.177464  5226 raft_consensus.cc:374] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:10.177556  5226 raft_consensus.cc:729] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Initialized, Role: FOLLOWER
I20240503 02:12:10.177762  5226 consensus_queue.cc:260] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:10.178242  5226 ts_tablet_manager.cc:1398] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:12:10.236539  4961 raft_consensus.cc:482] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:10.236763  4961 raft_consensus.cc:504] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:10.237287  4961 leader_election.cc:288] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:10.237968  4193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9898815c904092af0a6f238ed961bd" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f19d54f56849f495dc547de6a47ad2" is_pre_election: true
I20240503 02:12:10.238237  4193 raft_consensus.cc:2454] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 0.
I20240503 02:12:10.238759  3998 leader_election.cc:302] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c4af56cee8e546c384cf20f40deeacd7, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:10.239058  4961 raft_consensus.cc:2781] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:10.239157  4961 raft_consensus.cc:482] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:10.239272  4961 raft_consensus.cc:3037] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:10.241180  4961 raft_consensus.cc:504] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:10.241674  4961 leader_election.cc:288] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 election: Requested vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:10.242323  4193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9898815c904092af0a6f238ed961bd" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
I20240503 02:12:10.242587  4193 raft_consensus.cc:3037] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:10.244879  4193 raft_consensus.cc:2454] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 1.
I20240503 02:12:10.245290  3998 leader_election.cc:302] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c4af56cee8e546c384cf20f40deeacd7, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:10.245596  4961 raft_consensus.cc:2781] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:10.245764  4961 raft_consensus.cc:686] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Becoming Leader. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Running, Role: LEADER
I20240503 02:12:10.246029  4961 consensus_queue.cc:237] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:10.248628  3919 catalog_manager.cc:5484] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:10.443989  4961 raft_consensus.cc:482] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:10.444176  4961 raft_consensus.cc:504] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:10.444754  4961 leader_election.cc:288] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:10.445351  4193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15fafdc84fec4b46979c311f377fd312" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f19d54f56849f495dc547de6a47ad2" is_pre_election: true
I20240503 02:12:10.445590  4193 raft_consensus.cc:2454] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 0.
I20240503 02:12:10.446015  3998 leader_election.cc:302] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c4af56cee8e546c384cf20f40deeacd7, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:10.446323  4961 raft_consensus.cc:2781] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:10.446450  4961 raft_consensus.cc:482] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:10.446538  4961 raft_consensus.cc:3037] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:10.448908  4961 raft_consensus.cc:504] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:10.449458  4961 leader_election.cc:288] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 election: Requested vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:10.450026  4193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15fafdc84fec4b46979c311f377fd312" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
I20240503 02:12:10.450238  4193 raft_consensus.cc:3037] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:10.452174  4193 raft_consensus.cc:2454] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 1.
I20240503 02:12:10.452629  3998 leader_election.cc:302] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c4af56cee8e546c384cf20f40deeacd7, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:10.452963  4961 raft_consensus.cc:2781] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:10.453115  4961 raft_consensus.cc:686] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Becoming Leader. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Running, Role: LEADER
I20240503 02:12:10.453403  4961 consensus_queue.cc:237] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } }
I20240503 02:12:10.456077  3919 catalog_manager.cc:5484] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:10.500458  5100 raft_consensus.cc:482] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:10.500694  5100 raft_consensus.cc:504] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:10.501351  5100 leader_election.cc:288] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:10.501891  4193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27609e0e32a34efc915bbff6750f8234" candidate_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f19d54f56849f495dc547de6a47ad2" is_pre_election: true
I20240503 02:12:10.502091  4193 raft_consensus.cc:2454] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b812251ceb44fbe8a54fea04a7f8dfc in term 0.
I20240503 02:12:10.502472  4260 leader_election.cc:302] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:10.502748  5100 raft_consensus.cc:2781] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:10.502869  5100 raft_consensus.cc:482] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:10.502933  5100 raft_consensus.cc:3037] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:10.504858  5100 raft_consensus.cc:504] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:10.505298  5100 leader_election.cc:288] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 election: Requested vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:10.505885  4193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27609e0e32a34efc915bbff6750f8234" candidate_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
I20240503 02:12:10.506073  4193 raft_consensus.cc:3037] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:10.508109  4193 raft_consensus.cc:2454] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b812251ceb44fbe8a54fea04a7f8dfc in term 1.
I20240503 02:12:10.508590  4260 leader_election.cc:302] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:10.508961  5100 raft_consensus.cc:2781] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:10.509122  5100 raft_consensus.cc:686] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Becoming Leader. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Running, Role: LEADER
I20240503 02:12:10.509447  5100 consensus_queue.cc:237] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } }
I20240503 02:12:10.512271  3919 catalog_manager.cc:5484] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3). New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:10.795526  4961 consensus_queue.cc:1035] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:10.904481  5171 consensus_queue.cc:1035] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:10.960785  5239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:10.961138  5239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:10.968741  5239 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 02:12:11.110654  4193 raft_consensus.cc:1261] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:11.111274  5100 consensus_queue.cc:1035] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:11.157076  5284 mvcc.cc:205] Tried to move back new op lower bound from 7023420748222132224 to 7023420745768673280. Current Snapshot: MvccSnapshot[applied={T|T < 7023420748222132224}]

INSERT report
    rows total: 2000
    time total: 174.227 ms
  time per row: 0.0871136 ms
I20240503 02:12:11.293913   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.4:33659
--local_ip_for_outbound_sockets=127.0.145.4
--tserver_master_addrs=127.0.145.62:40001
--webserver_port=34341
--webserver_interface=127.0.145.4
--builtin_ntp_servers=127.0.145.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:12:11.659868  5292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:11.660192  5292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:11.660360  5292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:11.668347  5292 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:11.668498  5292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:11.668666  5292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.4
I20240503 02:12:11.678296  5292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.4:33659
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.145.4
--webserver_port=34341
--tserver_master_addrs=127.0.145.62:40001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.4
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:11.679769  5292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:11.681543  5292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:11.694790  5298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:11.695518  5300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:11.696270  5297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:11.697314  5292 server_base.cc:1034] running on GCE node
I20240503 02:12:11.697942  5292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:11.699003  5292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:11.700240  5292 hybrid_clock.cc:648] HybridClock initialized: now 1714702331700190 us; error 36 us; skew 500 ppm
I20240503 02:12:11.700621  5292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:11.703248  5292 webserver.cc:460] Webserver started at http://127.0.145.4:34341/ using document root <none> and password file <none>
I20240503 02:12:11.704067  5292 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:11.704197  5292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:11.709782  5292 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240503 02:12:11.712688  5306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:11.714056  5292 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:12:11.714293  5292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6ebeaff495e042c78067c5b18600dc98"
format_stamp: "Formatted at 2024-05-03 02:11:58 on dist-test-slave-gq9h"
I20240503 02:12:11.715215  5292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:11.734833  5292 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:11.735728  5292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:11.736034  5292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:11.737076  5292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:11.739078  5292 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:11.739176  5292 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:11.739295  5292 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:11.739341  5292 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:11.772197  5292 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.4:33659
I20240503 02:12:11.772228  5418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.4:33659 every 8 connection(s)
I20240503 02:12:11.773695  5292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:12:11.781184   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 5292
I20240503 02:12:11.781762   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.5:39125
--local_ip_for_outbound_sockets=127.0.145.5
--tserver_master_addrs=127.0.145.62:40001
--webserver_port=43361
--webserver_interface=127.0.145.5
--builtin_ntp_servers=127.0.145.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:11.787957  5419 heartbeater.cc:347] Connected to a master server at 127.0.145.62:40001
I20240503 02:12:11.788283  5419 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:11.789158  5419 heartbeater.cc:511] Master 127.0.145.62:40001 requested a full tablet report, sending...
I20240503 02:12:11.790971  3919 ts_manager.cc:194] Re-registered known tserver with Master: 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4:33659)
I20240503 02:12:11.791827  3919 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.4:55775
W20240503 02:12:12.138540  5423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:12.138895  5423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:12.139056  5423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:12.146732  5423 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:12.146900  5423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:12.147081  5423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.5
I20240503 02:12:12.156947  5423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.5:39125
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.145.5
--webserver_port=43361
--tserver_master_addrs=127.0.145.62:40001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.5
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:12.158334  5423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:12.160293  5423 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:12.173492  5428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:12.174149  5431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:12.175061  5429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:12.175916  5423 server_base.cc:1034] running on GCE node
I20240503 02:12:12.176574  5423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:12.177625  5423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:12.178826  5423 hybrid_clock.cc:648] HybridClock initialized: now 1714702332178764 us; error 55 us; skew 500 ppm
I20240503 02:12:12.179162  5423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:12.181529  5423 webserver.cc:460] Webserver started at http://127.0.145.5:43361/ using document root <none> and password file <none>
I20240503 02:12:12.182420  5423 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:12.182555  5423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:12.188330  5423 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240503 02:12:12.191340  5437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:12.192960  5423 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:12:12.193161  5423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "37c60eb66a6b45aaac4e0d7061572da6"
format_stamp: "Formatted at 2024-05-03 02:11:58 on dist-test-slave-gq9h"
I20240503 02:12:12.194105  5423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:12.226336  5423 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:12.227270  5423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:12.227568  5423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:12.228583  5423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:12.230262  5423 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:12.230370  5423 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:12.230504  5423 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:12.230566  5423 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:12.262559  5423 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.5:39125
I20240503 02:12:12.262632  5549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.5:39125 every 8 connection(s)
I20240503 02:12:12.264191  5423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:12:12.269686   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 5423
I20240503 02:12:12.270272   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.6:36423
--local_ip_for_outbound_sockets=127.0.145.6
--tserver_master_addrs=127.0.145.62:40001
--webserver_port=41177
--webserver_interface=127.0.145.6
--builtin_ntp_servers=127.0.145.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:12.278460  5550 heartbeater.cc:347] Connected to a master server at 127.0.145.62:40001
I20240503 02:12:12.278800  5550 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:12.279668  5550 heartbeater.cc:511] Master 127.0.145.62:40001 requested a full tablet report, sending...
I20240503 02:12:12.281196  3919 ts_manager.cc:194] Re-registered known tserver with Master: 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5:39125)
I20240503 02:12:12.282361  3919 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.5:40931
W20240503 02:12:12.624414  5553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:12.624716  5553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:12.624826  5553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:12.632428  5553 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:12.632567  5553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:12.632694  5553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.6
I20240503 02:12:12.642709  5553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.6:36423
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.145.6
--webserver_port=41177
--tserver_master_addrs=127.0.145.62:40001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.6
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:12.644114  5553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:12.645924  5553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:12.659188  5558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:12.659765  5561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:12.660684  5559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:12.661474  5553 server_base.cc:1034] running on GCE node
I20240503 02:12:12.662145  5553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:12.663237  5553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:12.664439  5553 hybrid_clock.cc:648] HybridClock initialized: now 1714702332664375 us; error 65 us; skew 500 ppm
I20240503 02:12:12.664772  5553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:12.667223  5553 webserver.cc:460] Webserver started at http://127.0.145.6:41177/ using document root <none> and password file <none>
I20240503 02:12:12.668015  5553 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:12.668154  5553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:12.673853  5553 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240503 02:12:12.676970  5567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:12.678388  5553 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:12:12.678565  5553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9c804fe67ee34e7fae5fab23db90f298"
format_stamp: "Formatted at 2024-05-03 02:11:59 on dist-test-slave-gq9h"
I20240503 02:12:12.679499  5553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:12.729600  5553 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:12.730588  5553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:12.730921  5553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:12.731892  5553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:12.733412  5553 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:12.733484  5553 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:12.733603  5553 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:12.733650  5553 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:12.765403  5553 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.6:36423
I20240503 02:12:12.765441  5679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.6:36423 every 8 connection(s)
I20240503 02:12:12.767033  5553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:12:12.777335   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 5553
I20240503 02:12:12.777989   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.7:35457
--local_ip_for_outbound_sockets=127.0.145.7
--tserver_master_addrs=127.0.145.62:40001
--webserver_port=36903
--webserver_interface=127.0.145.7
--builtin_ntp_servers=127.0.145.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:12.780148  5680 heartbeater.cc:347] Connected to a master server at 127.0.145.62:40001
I20240503 02:12:12.780519  5680 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:12.781574  5680 heartbeater.cc:511] Master 127.0.145.62:40001 requested a full tablet report, sending...
I20240503 02:12:12.783372  3919 ts_manager.cc:194] Re-registered known tserver with Master: 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6:36423)
I20240503 02:12:12.784552  3919 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.6:52279
I20240503 02:12:12.794683  5419 heartbeater.cc:503] Master 127.0.145.62:40001 was elected leader, sending a full tablet report...
W20240503 02:12:13.135143  5683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:13.135459  5683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:13.135566  5683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:13.144061  5683 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:13.144202  5683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:13.144376  5683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.7
I20240503 02:12:13.155643  5683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.7:35457
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.145.7
--webserver_port=36903
--tserver_master_addrs=127.0.145.62:40001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.7
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:13.157049  5683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:13.158910  5683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:13.172922  5689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:13.173498  5688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:13.174479  5691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:13.175498  5683 server_base.cc:1034] running on GCE node
I20240503 02:12:13.176237  5683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:13.177270  5683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:13.178467  5683 hybrid_clock.cc:648] HybridClock initialized: now 1714702333178404 us; error 60 us; skew 500 ppm
I20240503 02:12:13.178838  5683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:13.181362  5683 webserver.cc:460] Webserver started at http://127.0.145.7:36903/ using document root <none> and password file <none>
I20240503 02:12:13.182312  5683 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:13.182432  5683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:13.188339  5683 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.000s	sys 0.004s
I20240503 02:12:13.191334  5697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:13.192780  5683 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:12:13.192977  5683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6c3559ccff5d4c6681b3c90f892143dc"
format_stamp: "Formatted at 2024-05-03 02:11:59 on dist-test-slave-gq9h"
I20240503 02:12:13.193908  5683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:13.232991  5683 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:13.233868  5683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:13.234154  5683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:13.235211  5683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:13.236825  5683 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:13.236905  5683 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:13.237071  5683 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:13.237129  5683 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:13.272258  5683 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.7:35457
I20240503 02:12:13.272310  5809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.7:35457 every 8 connection(s)
I20240503 02:12:13.273844  5683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:12:13.275173   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 5683
I20240503 02:12:13.275665   580 test_util.cc:268] Using random seed: 1064857068
I20240503 02:12:13.284992  5550 heartbeater.cc:503] Master 127.0.145.62:40001 was elected leader, sending a full tablet report...
I20240503 02:12:13.288756  5810 heartbeater.cc:347] Connected to a master server at 127.0.145.62:40001
I20240503 02:12:13.289110  5810 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:13.290030  5810 heartbeater.cc:511] Master 127.0.145.62:40001 requested a full tablet report, sending...
I20240503 02:12:13.291466  3919 ts_manager.cc:194] Re-registered known tserver with Master: 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7:35457)
I20240503 02:12:13.292309  3919 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.7:40769
I20240503 02:12:13.294636   580 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 02:12:13.295217   580 test_util.cc:268] Using random seed: 1064876611
I20240503 02:12:13.312220   580 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 02:12:13.314910   580 test_util.cc:268] Using random seed: 1064896275
I20240503 02:12:13.366961   580 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 02:12:13.367478   580 test_util.cc:268] Using random seed: 1064948848
I20240503 02:12:13.441395   580 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 02:12:13.443742   580 test_util.cc:268] Using random seed: 1065025105
I20240503 02:12:13.555037   580 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240503 02:12:13.787825  5680 heartbeater.cc:503] Master 127.0.145.62:40001 was elected leader, sending a full tablet report...
I20240503 02:12:14.295352  5810 heartbeater.cc:503] Master 127.0.145.62:40001 was elected leader, sending a full tablet report...
I20240503 02:12:15.269464  5485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:15.310777  5745 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:15.315245  5354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:15.334242  5615 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:15.364701  4305 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:15.367040  4043 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:12:15.468436  4173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:16.161597  5485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:16.198518  5354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:16.204872  4305 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:16.227715  5745 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:16.299264  4043 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:12:16.304436  5615 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:16.334851  4173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:16.624352  4063 consensus_queue.cc:237] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } }
I20240503 02:12:16.630533  4194 raft_consensus.cc:1261] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 02:12:16.631489  5890 consensus_queue.cc:1035] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
W20240503 02:12:16.639429  3999 consensus_peers.cc:485] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 -> Peer 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7:35457): Couldn't send request to peer 6c3559ccff5d4c6681b3c90f892143dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15fafdc84fec4b46979c311f377fd312. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:16.650990  5924 raft_consensus.cc:2932] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index -1 to 59, NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } } }
I20240503 02:12:16.658079  3915 catalog_manager.cc:5484] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index -1 to 59, NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:16.658779  4194 raft_consensus.cc:2932] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index -1 to 59, NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } } }
I20240503 02:12:16.740447  4194 consensus_queue.cc:237] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 1.61, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } }
I20240503 02:12:16.747931  4322 raft_consensus.cc:1261] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 02:12:16.748687  5984 consensus_queue.cc:1035] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20240503 02:12:16.758993  5843 raft_consensus.cc:2932] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } } }
I20240503 02:12:16.768411  4325 raft_consensus.cc:2932] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } } }
I20240503 02:12:16.787581  3919 catalog_manager.cc:5484] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: config changed from index -1 to 62, NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:16.824103  4129 consensus_peers.cc:485] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 -> Peer 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5:39125): Couldn't send request to peer 37c60eb66a6b45aaac4e0d7061572da6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed79dde0a1f4419c8085c042ed5b8780. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:16.871610  4063 consensus_queue.cc:237] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 1.61, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } }
I20240503 02:12:16.891556  4192 raft_consensus.cc:1261] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
I20240503 02:12:16.893127  5924 consensus_queue.cc:1035] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20240503 02:12:16.900189  5890 raft_consensus.cc:2932] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } } }
I20240503 02:12:16.907981  3919 catalog_manager.cc:5484] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index -1 to 62, NON_VOTER 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) added. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:16.916582  4192 raft_consensus.cc:2932] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } } }
W20240503 02:12:16.921191  3999 consensus_peers.cc:485] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 -> Peer 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4:33659): Couldn't send request to peer 6ebeaff495e042c78067c5b18600dc98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e9898815c904092af0a6f238ed961bd. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:17.047402  4063 consensus_queue.cc:237] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
I20240503 02:12:17.064431  4192 raft_consensus.cc:1261] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 02:12:17.065488  5926 consensus_queue.cc:1035] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 02:12:17.070978  5890 raft_consensus.cc:2932] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } } }
I20240503 02:12:17.071720  4194 raft_consensus.cc:2932] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } } }
I20240503 02:12:17.076478  3919 catalog_manager.cc:5484] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index -1 to 29, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:17.092062  3998 consensus_peers.cc:485] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 -> Peer 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6:36423): Couldn't send request to peer 9c804fe67ee34e7fae5fab23db90f298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d846ad6206514c79b142064043ad1cb0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:17.110795  6047 ts_tablet_manager.cc:902] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc: Initiating tablet copy from peer c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:17.138952  6055 ts_tablet_manager.cc:902] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6: Initiating tablet copy from peer d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:17.139551  6047 tablet_copy_client.cc:308] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:43835
I20240503 02:12:17.174090  6055 tablet_copy_client.cc:308] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:44433
I20240503 02:12:17.181838  4083 tablet_copy_service.cc:140] P c4af56cee8e546c384cf20f40deeacd7: Received BeginTabletCopySession request for tablet 15fafdc84fec4b46979c311f377fd312 from peer 6c3559ccff5d4c6681b3c90f892143dc ({username='slave'} at 127.0.145.7:41467)
I20240503 02:12:17.182179  4083 tablet_copy_service.cc:161] P c4af56cee8e546c384cf20f40deeacd7: Beginning new tablet copy session on tablet 15fafdc84fec4b46979c311f377fd312 from peer 6c3559ccff5d4c6681b3c90f892143dc at {username='slave'} at 127.0.145.7:41467: session id = 6c3559ccff5d4c6681b3c90f892143dc-15fafdc84fec4b46979c311f377fd312
I20240503 02:12:17.187938  4214 tablet_copy_service.cc:140] P d7f19d54f56849f495dc547de6a47ad2: Received BeginTabletCopySession request for tablet ed79dde0a1f4419c8085c042ed5b8780 from peer 37c60eb66a6b45aaac4e0d7061572da6 ({username='slave'} at 127.0.145.5:43341)
I20240503 02:12:17.188269  4214 tablet_copy_service.cc:161] P d7f19d54f56849f495dc547de6a47ad2: Beginning new tablet copy session on tablet ed79dde0a1f4419c8085c042ed5b8780 from peer 37c60eb66a6b45aaac4e0d7061572da6 at {username='slave'} at 127.0.145.5:43341: session id = 37c60eb66a6b45aaac4e0d7061572da6-ed79dde0a1f4419c8085c042ed5b8780
I20240503 02:12:17.190272  4083 tablet_copy_source_session.cc:205] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:17.195128  4214 tablet_copy_source_session.cc:205] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:17.199548  6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15fafdc84fec4b46979c311f377fd312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.200268  4324 consensus_queue.cc:237] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 1.88, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } }
I20240503 02:12:17.203171  4063 raft_consensus.cc:1261] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch)
I20240503 02:12:17.203749  6055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed79dde0a1f4419c8085c042ed5b8780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.203913  5862 consensus_queue.cc:1035] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20240503 02:12:17.218307  5862 raft_consensus.cc:2932] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.89: config changed from index -1 to 89, NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) added. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } } }
I20240503 02:12:17.227870  4061 raft_consensus.cc:2932] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index -1 to 89, NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) added. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } } }
I20240503 02:12:17.230446  3915 catalog_manager.cc:5484] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index -1 to 89, NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:17.237174  4261 consensus_peers.cc:485] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc -> Peer 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7:35457): Couldn't send request to peer 6c3559ccff5d4c6681b3c90f892143dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df156172a4554094b36cd5cc3876c922. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:17.242842  6055 tablet_copy_client.cc:783] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:17.243227  6055 tablet_copy_client.cc:647] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:17.252039  6055 tablet_copy_client.cc:523] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:17.258131  6055 tablet_bootstrap.cc:492] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6: Bootstrap starting.
I20240503 02:12:17.295428  6047 tablet_copy_client.cc:783] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:17.301339  6047 tablet_copy_client.cc:647] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:17.307606  6055 log.cc:824] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:17.328826  6047 tablet_copy_client.cc:523] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:17.353729  6047 tablet_bootstrap.cc:492] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc: Bootstrap starting.
I20240503 02:12:17.392786  6047 log.cc:824] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:17.418449  4063 consensus_queue.cc:237] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } }
I20240503 02:12:17.451294  4324 raft_consensus.cc:1261] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240503 02:12:17.464047  5890 consensus_queue.cc:1035] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240503 02:12:17.469399  6074 ts_tablet_manager.cc:902] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98: Initiating tablet copy from peer c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
W20240503 02:12:17.482110  3998 consensus_peers.cc:485] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 -> Peer 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5:39125): Couldn't send request to peer 37c60eb66a6b45aaac4e0d7061572da6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b76c95bac0f497f8ab21c5295028026. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:17.494563  5926 raft_consensus.cc:2932] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } } }
I20240503 02:12:17.502939  3919 catalog_manager.cc:5484] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index -1 to 84, NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:17.539206  6047 tablet_bootstrap.cc:492] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=66 overwritten=0 applied=66 ignored=0} inserts{seen=738 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:17.540371  6047 tablet_bootstrap.cc:492] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc: Bootstrap complete.
I20240503 02:12:17.540232  4322 raft_consensus.cc:2932] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } } }
I20240503 02:12:17.541172  6047 ts_tablet_manager.cc:1367] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc: Time spent bootstrapping tablet: real 0.188s	user 0.123s	sys 0.034s
I20240503 02:12:17.547984  6047 raft_consensus.cc:348] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } }
I20240503 02:12:17.548765  6047 raft_consensus.cc:729] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c3559ccff5d4c6681b3c90f892143dc, State: Initialized, Role: LEARNER
I20240503 02:12:17.549511  6047 consensus_queue.cc:260] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 66, Last appended: 1.66, Last appended by leader: 66, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } }
I20240503 02:12:17.558830  6047 ts_tablet_manager.cc:1398] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc: Time spent starting tablet: real 0.017s	user 0.016s	sys 0.000s
I20240503 02:12:17.559980  4083 tablet_copy_service.cc:342] P c4af56cee8e546c384cf20f40deeacd7: Request end of tablet copy session 6c3559ccff5d4c6681b3c90f892143dc-15fafdc84fec4b46979c311f377fd312 received from {username='slave'} at 127.0.145.7:41467
I20240503 02:12:17.560178  4083 tablet_copy_service.cc:433] P c4af56cee8e546c384cf20f40deeacd7: ending tablet copy session 6c3559ccff5d4c6681b3c90f892143dc-15fafdc84fec4b46979c311f377fd312 on tablet 15fafdc84fec4b46979c311f377fd312 with peer 6c3559ccff5d4c6681b3c90f892143dc
I20240503 02:12:17.572548  6055 tablet_bootstrap.cc:492] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=69 ignored=0} inserts{seen=700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:12:17.573254  6055 tablet_bootstrap.cc:492] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6: Bootstrap complete.
I20240503 02:12:17.573807  6055 ts_tablet_manager.cc:1367] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6: Time spent bootstrapping tablet: real 0.316s	user 0.138s	sys 0.006s
I20240503 02:12:17.600984  6055 raft_consensus.cc:348] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } }
I20240503 02:12:17.608100  6055 raft_consensus.cc:729] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 37c60eb66a6b45aaac4e0d7061572da6, State: Initialized, Role: LEARNER
I20240503 02:12:17.609408  6055 consensus_queue.cc:260] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } }
I20240503 02:12:17.613466  6055 ts_tablet_manager.cc:1398] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6: Time spent starting tablet: real 0.040s	user 0.015s	sys 0.000s
I20240503 02:12:17.614564  4214 tablet_copy_service.cc:342] P d7f19d54f56849f495dc547de6a47ad2: Request end of tablet copy session 37c60eb66a6b45aaac4e0d7061572da6-ed79dde0a1f4419c8085c042ed5b8780 received from {username='slave'} at 127.0.145.5:43341
I20240503 02:12:17.614786  4214 tablet_copy_service.cc:433] P d7f19d54f56849f495dc547de6a47ad2: ending tablet copy session 37c60eb66a6b45aaac4e0d7061572da6-ed79dde0a1f4419c8085c042ed5b8780 on tablet ed79dde0a1f4419c8085c042ed5b8780 with peer 37c60eb66a6b45aaac4e0d7061572da6
I20240503 02:12:17.618988  6083 ts_tablet_manager.cc:902] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: Initiating tablet copy from peer c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:17.681396  5505 raft_consensus.cc:1203] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.61->[1.62-1.78]   Dedup: 1.70->[1.71-1.78]
I20240503 02:12:17.684728  6047 ts_tablet_manager.cc:902] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc: Initiating tablet copy from peer 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:17.774400  6083 tablet_copy_client.cc:308] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:43835
I20240503 02:12:17.803525  6047 tablet_copy_client.cc:308] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:41599
I20240503 02:12:17.832731  4083 tablet_copy_service.cc:140] P c4af56cee8e546c384cf20f40deeacd7: Received BeginTabletCopySession request for tablet d846ad6206514c79b142064043ad1cb0 from peer 9c804fe67ee34e7fae5fab23db90f298 ({username='slave'} at 127.0.145.6:39277)
I20240503 02:12:17.833158  4083 tablet_copy_service.cc:161] P c4af56cee8e546c384cf20f40deeacd7: Beginning new tablet copy session on tablet d846ad6206514c79b142064043ad1cb0 from peer 9c804fe67ee34e7fae5fab23db90f298 at {username='slave'} at 127.0.145.6:39277: session id = 9c804fe67ee34e7fae5fab23db90f298-d846ad6206514c79b142064043ad1cb0
I20240503 02:12:17.836324  4083 tablet_copy_source_session.cc:205] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:17.842182  4325 consensus_queue.cc:237] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.48, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } }
I20240503 02:12:17.845285  6083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d846ad6206514c79b142064043ad1cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.846112  5843 raft_consensus.cc:1050] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2: attempting to promote NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 to VOTER
I20240503 02:12:17.847146  5843 consensus_queue.cc:237] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } }
I20240503 02:12:17.854229  5505 raft_consensus.cc:1261] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240503 02:12:17.855151  5843 consensus_queue.cc:1035] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 79, Time since last communication: 0.000s
I20240503 02:12:17.861115  4324 raft_consensus.cc:1261] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240503 02:12:17.862134  4193 raft_consensus.cc:1261] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240503 02:12:17.861775  6031 raft_consensus.cc:2932] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index 62 to 81, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:17.863778  4345 tablet_copy_service.cc:140] P 8b812251ceb44fbe8a54fea04a7f8dfc: Received BeginTabletCopySession request for tablet df156172a4554094b36cd5cc3876c922 from peer 6c3559ccff5d4c6681b3c90f892143dc ({username='slave'} at 127.0.145.7:50777)
I20240503 02:12:17.864020  4345 tablet_copy_service.cc:161] P 8b812251ceb44fbe8a54fea04a7f8dfc: Beginning new tablet copy session on tablet df156172a4554094b36cd5cc3876c922 from peer 6c3559ccff5d4c6681b3c90f892143dc at {username='slave'} at 127.0.145.7:50777: session id = 6c3559ccff5d4c6681b3c90f892143dc-df156172a4554094b36cd5cc3876c922
I20240503 02:12:17.867787  5984 consensus_queue.cc:1035] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 79, Time since last communication: 0.000s
I20240503 02:12:17.872469  3915 catalog_manager.cc:5484] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: config changed from index 62 to 81, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:17.872752  5504 raft_consensus.cc:2932] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index 62 to 81, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:17.876595  4345 tablet_copy_source_session.cc:205] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:17.879367  6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df156172a4554094b36cd5cc3876c922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:17.883277  4324 raft_consensus.cc:2932] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index 62 to 81, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:17.886941  6047 tablet_copy_client.cc:783] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:17.887234  6047 tablet_copy_client.cc:647] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:17.888208  6074 tablet_copy_client.cc:308] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:43835
I20240503 02:12:17.890208  4194 consensus_queue.cc:237] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } }
I20240503 02:12:17.891918  5860 consensus_queue.cc:1035] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 47, Time since last communication: 0.000s
I20240503 02:12:17.893818  6083 tablet_copy_client.cc:783] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:17.894146  5505 raft_consensus.cc:1261] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240503 02:12:17.894201  6083 tablet_copy_client.cc:647] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:17.895115  5843 consensus_queue.cc:1035] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20240503 02:12:17.896232  4194 raft_consensus.cc:1203] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.47->[1.48-1.49]   Dedup: 1.48->[1.49-1.49]
W20240503 02:12:17.898097  4261 consensus_peers.cc:485] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc -> Peer 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4:33659): Couldn't send request to peer 6ebeaff495e042c78067c5b18600dc98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec5c9e506c2f4c29aea9a9de259a0c19. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:17.898594  6083 tablet_copy_client.cc:523] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:17.908056  6083 tablet_bootstrap.cc:492] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap starting.
I20240503 02:12:17.913349  6047 tablet_copy_client.cc:523] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:17.915304  5984 raft_consensus.cc:2932] T ed79dde0a1f4419c8085c042ed5b8780 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index 81 to 83, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:17.917124  6047 tablet_bootstrap.cc:492] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc: Bootstrap starting.
I20240503 02:12:17.922333  5504 raft_consensus.cc:2932] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 81 to 83, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:17.925855  5943 raft_consensus.cc:2932] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.49: config changed from index -1 to 49, NON_VOTER 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) added. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } } }
I20240503 02:12:17.927737  3908 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ed79dde0a1f4419c8085c042ed5b8780 with cas_config_opid_index 81: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:17.939249  3919 catalog_manager.cc:5484] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index -1 to 49, NON_VOTER 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) added. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:17.943109  3918 catalog_manager.cc:5484] T ed79dde0a1f4419c8085c042ed5b8780 P 37c60eb66a6b45aaac4e0d7061572da6 reported cstate change: config changed from index 81 to 83, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:17.948459  4305 tablet_service.cc:1514] Processing DeleteTablet for tablet ed79dde0a1f4419c8085c042ed5b8780 with delete_type TABLET_DATA_TOMBSTONED (TS 8b812251ceb44fbe8a54fea04a7f8dfc not found in new config with opid_index 83) from {username='slave'} at 127.0.0.1:54992
I20240503 02:12:17.976071  6111 tablet_replica.cc:330] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc: stopping tablet replica
I20240503 02:12:17.977008  6111 raft_consensus.cc:2229] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:17.978418  6111 raft_consensus.cc:2258] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:17.986349  4193 raft_consensus.cc:2932] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index -1 to 49, NON_VOTER 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) added. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } } }
I20240503 02:12:17.988063  6083 log.cc:824] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:17.996112  6055 ts_tablet_manager.cc:902] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6: Initiating tablet copy from peer c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:17.997944  6111 ts_tablet_manager.cc:1875] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:18.007200  6111 ts_tablet_manager.cc:1888] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.81
I20240503 02:12:18.007432  6111 log.cc:1196] T ed79dde0a1f4419c8085c042ed5b8780 P 8b812251ceb44fbe8a54fea04a7f8dfc: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/ed79dde0a1f4419c8085c042ed5b8780
I20240503 02:12:18.008571  3907 catalog_manager.cc:4830] TS 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599): tablet ed79dde0a1f4419c8085c042ed5b8780 (table rebalance_test_table_1 [id=908c33982a334c0a957523734a0931d3]) successfully deleted
I20240503 02:12:18.017843  4083 tablet_copy_service.cc:140] P c4af56cee8e546c384cf20f40deeacd7: Received BeginTabletCopySession request for tablet 6e9898815c904092af0a6f238ed961bd from peer 6ebeaff495e042c78067c5b18600dc98 ({username='slave'} at 127.0.145.4:41257)
I20240503 02:12:18.018043  4083 tablet_copy_service.cc:161] P c4af56cee8e546c384cf20f40deeacd7: Beginning new tablet copy session on tablet 6e9898815c904092af0a6f238ed961bd from peer 6ebeaff495e042c78067c5b18600dc98 at {username='slave'} at 127.0.145.4:41257: session id = 6ebeaff495e042c78067c5b18600dc98-6e9898815c904092af0a6f238ed961bd
I20240503 02:12:18.024551  4083 tablet_copy_source_session.cc:205] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:18.030642  6074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9898815c904092af0a6f238ed961bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.078727  6074 tablet_copy_client.cc:783] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:18.079193  6074 tablet_copy_client.cc:647] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:18.086803  6074 tablet_copy_client.cc:523] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:18.091555  6074 tablet_bootstrap.cc:492] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98: Bootstrap starting.
I20240503 02:12:18.094398  6055 tablet_copy_client.cc:308] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:43835
I20240503 02:12:18.093973  4322 consensus_queue.cc:237] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.101, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
I20240503 02:12:18.099155  4061 raft_consensus.cc:1261] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240503 02:12:18.099907  6038 consensus_queue.cc:1035] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240503 02:12:18.103965  5862 raft_consensus.cc:2932] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.102: config changed from index -1 to 102, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } } }
I20240503 02:12:18.116514  3915 catalog_manager.cc:5484] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index -1 to 102, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:18.120183  4062 raft_consensus.cc:2932] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index -1 to 102, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } } }
I20240503 02:12:18.128244  5765 raft_consensus.cc:1203] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.58->[1.59-1.78]   Dedup: 1.66->[1.67-1.78]
I20240503 02:12:18.144855  4083 tablet_copy_service.cc:140] P c4af56cee8e546c384cf20f40deeacd7: Received BeginTabletCopySession request for tablet 8b76c95bac0f497f8ab21c5295028026 from peer 37c60eb66a6b45aaac4e0d7061572da6 ({username='slave'} at 127.0.145.5:47817)
I20240503 02:12:18.145102  4083 tablet_copy_service.cc:161] P c4af56cee8e546c384cf20f40deeacd7: Beginning new tablet copy session on tablet 8b76c95bac0f497f8ab21c5295028026 from peer 37c60eb66a6b45aaac4e0d7061572da6 at {username='slave'} at 127.0.145.5:47817: session id = 37c60eb66a6b45aaac4e0d7061572da6-8b76c95bac0f497f8ab21c5295028026
I20240503 02:12:18.148409  4083 tablet_copy_source_session.cc:205] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:18.171445  6055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b76c95bac0f497f8ab21c5295028026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.179183  6055 tablet_copy_client.cc:783] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:18.179473  6055 tablet_copy_client.cc:647] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:18.195456  6055 tablet_copy_client.cc:523] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:18.199518  6055 tablet_bootstrap.cc:492] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6: Bootstrap starting.
I20240503 02:12:18.251670  6074 log.cc:824] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:18.343937  6083 tablet_bootstrap.cc:492] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=706 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:18.344652  6083 tablet_bootstrap.cc:492] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap complete.
I20240503 02:12:18.345167  6083 ts_tablet_manager.cc:1367] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: Time spent bootstrapping tablet: real 0.437s	user 0.105s	sys 0.000s
I20240503 02:12:18.358942  6083 raft_consensus.cc:348] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
I20240503 02:12:18.359503  6083 raft_consensus.cc:729] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9c804fe67ee34e7fae5fab23db90f298, State: Initialized, Role: LEARNER
I20240503 02:12:18.363307  6083 consensus_queue.cc:260] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 1.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
I20240503 02:12:18.377336  6083 ts_tablet_manager.cc:1398] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: Time spent starting tablet: real 0.032s	user 0.016s	sys 0.000s
I20240503 02:12:18.379261  4083 tablet_copy_service.cc:342] P c4af56cee8e546c384cf20f40deeacd7: Request end of tablet copy session 9c804fe67ee34e7fae5fab23db90f298-d846ad6206514c79b142064043ad1cb0 received from {username='slave'} at 127.0.145.6:39277
I20240503 02:12:18.379418  4083 tablet_copy_service.cc:433] P c4af56cee8e546c384cf20f40deeacd7: ending tablet copy session 9c804fe67ee34e7fae5fab23db90f298-d846ad6206514c79b142064043ad1cb0 on tablet d846ad6206514c79b142064043ad1cb0 with peer 9c804fe67ee34e7fae5fab23db90f298
I20240503 02:12:18.393139  4193 consensus_queue.cc:237] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 1.109, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } }
I20240503 02:12:18.409349  6147 ts_tablet_manager.cc:902] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98: Initiating tablet copy from peer 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:18.409706  4062 raft_consensus.cc:1261] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20240503 02:12:18.412637  5843 consensus_queue.cc:1035] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
W20240503 02:12:18.427492  4260 consensus_peers.cc:485] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc -> Peer 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6:36423): Couldn't send request to peer 9c804fe67ee34e7fae5fab23db90f298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8334f743021f4957a427800af656bec1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:18.451208  5635 raft_consensus.cc:1203] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.37]   Dedup: 1.34->[1.35-1.37]
I20240503 02:12:18.454118  5924 raft_consensus.cc:1050] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7: attempting to promote NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc to VOTER
I20240503 02:12:18.455268  5924 consensus_queue.cc:237] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:18.458487  5764 raft_consensus.cc:1261] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240503 02:12:18.459695  6130 consensus_queue.cc:1035] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 79, Time since last communication: 0.000s
I20240503 02:12:18.465315  6130 raft_consensus.cc:2932] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index 59 to 81, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:18.473506  3915 catalog_manager.cc:5484] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index 59 to 81, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:18.479851  6047 tablet_bootstrap.cc:492] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=93 overwritten=0 applied=93 ignored=0} inserts{seen=765 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:18.479841  4061 consensus_queue.cc:237] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:18.481577  6047 tablet_bootstrap.cc:492] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc: Bootstrap complete.
I20240503 02:12:18.483336  5764 raft_consensus.cc:1261] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240503 02:12:18.483534  6047 ts_tablet_manager.cc:1367] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc: Time spent bootstrapping tablet: real 0.567s	user 0.163s	sys 0.012s
I20240503 02:12:18.484180  5890 consensus_queue.cc:1035] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 80, Time since last communication: 0.000s
I20240503 02:12:18.485219  6047 raft_consensus.cc:348] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } }
I20240503 02:12:18.489235  6047 raft_consensus.cc:729] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c3559ccff5d4c6681b3c90f892143dc, State: Initialized, Role: LEARNER
I20240503 02:12:18.485684  5764 raft_consensus.cc:2932] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index 59 to 81, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:18.490813  4193 raft_consensus.cc:1261] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
W20240503 02:12:18.491629  4130 consensus_peers.cc:485] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 -> Peer 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7:35457): Couldn't send request to peer 6c3559ccff5d4c6681b3c90f892143dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75a2da7ca68b43e28cad0847e4d27982. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:18.493100  6047 consensus_queue.cc:260] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 93, Last appended: 1.93, Last appended by leader: 93, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } }
I20240503 02:12:18.494725  6047 ts_tablet_manager.cc:1398] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.000s
I20240503 02:12:18.492201  6031 raft_consensus.cc:2932] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Committing config change with OpId 1.110: config changed from index -1 to 110, NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } } }
I20240503 02:12:18.503612  4062 raft_consensus.cc:2932] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index -1 to 110, NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } } }
I20240503 02:12:18.504690  3919 catalog_manager.cc:5484] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: config changed from index -1 to 110, NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:18.519767  5890 raft_consensus.cc:2932] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index 81 to 82, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:18.524073  3909 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 15fafdc84fec4b46979c311f377fd312 with cas_config_opid_index 81: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:18.533139  3918 catalog_manager.cc:5484] T 15fafdc84fec4b46979c311f377fd312 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index 81 to 82, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:18.533934  6055 tablet_bootstrap.cc:492] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6: Bootstrap replayed 1/1 log segments. Stats: ops{read=88 overwritten=0 applied=88 ignored=0} inserts{seen=760 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:18.534343  6055 tablet_bootstrap.cc:492] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6: Bootstrap complete.
I20240503 02:12:18.534680  6055 ts_tablet_manager.cc:1367] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6: Time spent bootstrapping tablet: real 0.336s	user 0.129s	sys 0.026s
I20240503 02:12:18.535809  6055 raft_consensus.cc:348] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } }
I20240503 02:12:18.536166  6055 raft_consensus.cc:729] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 37c60eb66a6b45aaac4e0d7061572da6, State: Initialized, Role: LEARNER
I20240503 02:12:18.536439  6055 consensus_queue.cc:260] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 88, Last appended: 1.88, Last appended by leader: 88, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } }
I20240503 02:12:18.537926  4173 tablet_service.cc:1514] Processing DeleteTablet for tablet 15fafdc84fec4b46979c311f377fd312 with delete_type TABLET_DATA_TOMBSTONED (TS d7f19d54f56849f495dc547de6a47ad2 not found in new config with opid_index 82) from {username='slave'} at 127.0.0.1:55800
I20240503 02:12:18.547129  5764 raft_consensus.cc:2932] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 81 to 82, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:18.579113  6074 tablet_bootstrap.cc:492] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98: Bootstrap replayed 1/1 log segments. Stats: ops{read=83 overwritten=0 applied=82 ignored=0} inserts{seen=767 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:12:18.580011  6074 tablet_bootstrap.cc:492] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98: Bootstrap complete.
I20240503 02:12:18.580636  6074 ts_tablet_manager.cc:1367] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98: Time spent bootstrapping tablet: real 0.489s	user 0.166s	sys 0.006s
I20240503 02:12:18.584785  4345 tablet_copy_service.cc:342] P 8b812251ceb44fbe8a54fea04a7f8dfc: Request end of tablet copy session 6c3559ccff5d4c6681b3c90f892143dc-df156172a4554094b36cd5cc3876c922 received from {username='slave'} at 127.0.145.7:50777
I20240503 02:12:18.585012  4345 tablet_copy_service.cc:433] P 8b812251ceb44fbe8a54fea04a7f8dfc: ending tablet copy session 6c3559ccff5d4c6681b3c90f892143dc-df156172a4554094b36cd5cc3876c922 on tablet df156172a4554094b36cd5cc3876c922 with peer 6c3559ccff5d4c6681b3c90f892143dc
I20240503 02:12:18.587791  6074 raft_consensus.cc:348] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } }
I20240503 02:12:18.597899  6074 raft_consensus.cc:729] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ebeaff495e042c78067c5b18600dc98, State: Initialized, Role: LEARNER
I20240503 02:12:18.599004  6074 consensus_queue.cc:260] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 82, Last appended: 1.83, Last appended by leader: 83, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } }
I20240503 02:12:18.607978  6074 ts_tablet_manager.cc:1398] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98: Time spent starting tablet: real 0.027s	user 0.016s	sys 0.008s
I20240503 02:12:18.609099  4083 tablet_copy_service.cc:342] P c4af56cee8e546c384cf20f40deeacd7: Request end of tablet copy session 6ebeaff495e042c78067c5b18600dc98-6e9898815c904092af0a6f238ed961bd received from {username='slave'} at 127.0.145.4:41257
I20240503 02:12:18.609257  4083 tablet_copy_service.cc:433] P c4af56cee8e546c384cf20f40deeacd7: ending tablet copy session 6ebeaff495e042c78067c5b18600dc98-6e9898815c904092af0a6f238ed961bd on tablet 6e9898815c904092af0a6f238ed961bd with peer 6ebeaff495e042c78067c5b18600dc98
I20240503 02:12:18.611891  6162 tablet_replica.cc:330] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2: stopping tablet replica
I20240503 02:12:18.612434  6162 raft_consensus.cc:2229] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.612838  6162 raft_consensus.cc:2258] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.613498  6147 tablet_copy_client.cc:308] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:41599
I20240503 02:12:18.616631  6162 ts_tablet_manager.cc:1875] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:18.622503  6055 ts_tablet_manager.cc:1398] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6: Time spent starting tablet: real 0.088s	user 0.031s	sys 0.030s
I20240503 02:12:18.626127  6162 ts_tablet_manager.cc:1888] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.80
I20240503 02:12:18.626323  6162 log.cc:1196] T 15fafdc84fec4b46979c311f377fd312 P d7f19d54f56849f495dc547de6a47ad2: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/15fafdc84fec4b46979c311f377fd312
I20240503 02:12:18.627418  3908 catalog_manager.cc:4830] TS d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433): tablet 15fafdc84fec4b46979c311f377fd312 (table rebalance_test_table_4 [id=7bd39e7281604b6ebd6334a68999befc]) successfully deleted
I20240503 02:12:18.630234  4083 tablet_copy_service.cc:342] P c4af56cee8e546c384cf20f40deeacd7: Request end of tablet copy session 37c60eb66a6b45aaac4e0d7061572da6-8b76c95bac0f497f8ab21c5295028026 received from {username='slave'} at 127.0.145.5:47817
I20240503 02:12:18.630394  4083 tablet_copy_service.cc:433] P c4af56cee8e546c384cf20f40deeacd7: ending tablet copy session 37c60eb66a6b45aaac4e0d7061572da6-8b76c95bac0f497f8ab21c5295028026 on tablet 8b76c95bac0f497f8ab21c5295028026 with peer 37c60eb66a6b45aaac4e0d7061572da6
I20240503 02:12:18.631570  4324 consensus_queue.cc:237] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } }
I20240503 02:12:18.631769  6083 ts_tablet_manager.cc:902] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298: Initiating tablet copy from peer 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:18.688649  4345 tablet_copy_service.cc:140] P 8b812251ceb44fbe8a54fea04a7f8dfc: Received BeginTabletCopySession request for tablet ec5c9e506c2f4c29aea9a9de259a0c19 from peer 6ebeaff495e042c78067c5b18600dc98 ({username='slave'} at 127.0.145.4:52377)
I20240503 02:12:18.688923  4345 tablet_copy_service.cc:161] P 8b812251ceb44fbe8a54fea04a7f8dfc: Beginning new tablet copy session on tablet ec5c9e506c2f4c29aea9a9de259a0c19 from peer 6ebeaff495e042c78067c5b18600dc98 at {username='slave'} at 127.0.145.4:52377: session id = 6ebeaff495e042c78067c5b18600dc98-ec5c9e506c2f4c29aea9a9de259a0c19
I20240503 02:12:18.690037  6130 raft_consensus.cc:1050] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7: attempting to promote NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 to VOTER
I20240503 02:12:18.691061  6130 consensus_queue.cc:237] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } }
I20240503 02:12:18.692401  4345 tablet_copy_source_session.cc:205] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:18.694519  6147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec5c9e506c2f4c29aea9a9de259a0c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.696700  5635 raft_consensus.cc:1261] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240503 02:12:18.697501  4194 raft_consensus.cc:1261] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240503 02:12:18.698398  6079 consensus_queue.cc:1035] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 37, Time since last communication: 0.000s
I20240503 02:12:18.698971  6079 consensus_queue.cc:1035] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 37, Time since last communication: 0.000s
I20240503 02:12:18.701005  6147 tablet_copy_client.cc:783] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:18.701287  6147 tablet_copy_client.cc:647] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:18.704034  6147 tablet_copy_client.cc:523] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:18.705539  5890 raft_consensus.cc:2932] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index 29 to 39, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:18.708976  6147 tablet_bootstrap.cc:492] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98: Bootstrap starting.
I20240503 02:12:18.714388  4194 raft_consensus.cc:2932] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 29 to 39, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:18.723472  5764 raft_consensus.cc:1203] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.88->[1.89-1.97]   Dedup: 1.93->[1.94-1.97]
W20240503 02:12:18.727605  4129 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.5:39125, user_credentials={real_user=slave}} blocked reactor thread for 53261.3us
I20240503 02:12:18.732422  5635 raft_consensus.cc:2932] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index 29 to 39, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:18.733731  4192 raft_consensus.cc:1261] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:18.734491  6124 consensus_queue.cc:1035] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:18.739801  3918 catalog_manager.cc:5484] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index 29 to 39, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:18.745935  6083 tablet_copy_client.cc:308] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:41599
I20240503 02:12:18.746251  4062 consensus_queue.cc:237] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } }
I20240503 02:12:18.752687  6124 raft_consensus.cc:2932] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } } }
I20240503 02:12:18.753201  4192 raft_consensus.cc:2932] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } } }
I20240503 02:12:18.766536  3920 catalog_manager.cc:5484] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index -1 to 2, NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:18.803107  6047 ts_tablet_manager.cc:902] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc: Initiating tablet copy from peer d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:18.804625  6047 tablet_copy_client.cc:308] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:44433
I20240503 02:12:18.807802  5634 raft_consensus.cc:1261] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20240503 02:12:18.808995  5890 consensus_queue.cc:1035] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 39, Time since last communication: 0.000s
I20240503 02:12:18.813565  6079 raft_consensus.cc:2932] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index 39 to 41, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:18.817579  3909 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d846ad6206514c79b142064043ad1cb0 with cas_config_opid_index 39: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:18.821099  3920 catalog_manager.cc:5484] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index 39 to 41, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:18.823948  5634 raft_consensus.cc:2932] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index 39 to 41, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:18.827586  4214 tablet_copy_service.cc:140] P d7f19d54f56849f495dc547de6a47ad2: Received BeginTabletCopySession request for tablet 75a2da7ca68b43e28cad0847e4d27982 from peer 6c3559ccff5d4c6681b3c90f892143dc ({username='slave'} at 127.0.145.7:42803)
I20240503 02:12:18.827800  4214 tablet_copy_service.cc:161] P d7f19d54f56849f495dc547de6a47ad2: Beginning new tablet copy session on tablet 75a2da7ca68b43e28cad0847e4d27982 from peer 6c3559ccff5d4c6681b3c90f892143dc at {username='slave'} at 127.0.145.7:42803: session id = 6c3559ccff5d4c6681b3c90f892143dc-75a2da7ca68b43e28cad0847e4d27982
I20240503 02:12:18.830732  4214 tablet_copy_source_session.cc:205] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:18.831110  4173 tablet_service.cc:1514] Processing DeleteTablet for tablet d846ad6206514c79b142064043ad1cb0 with delete_type TABLET_DATA_TOMBSTONED (TS d7f19d54f56849f495dc547de6a47ad2 not found in new config with opid_index 41) from {username='slave'} at 127.0.0.1:55800
I20240503 02:12:18.831449  6162 tablet_replica.cc:330] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2: stopping tablet replica
I20240503 02:12:18.831631  6162 raft_consensus.cc:2229] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.831796  6162 pending_rounds.cc:62] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2: Trying to abort 1 pending ops.
I20240503 02:12:18.831897  6162 pending_rounds.cc:68] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2: Aborting op as it isn't in flight: id { term: 1 index: 40 } timestamp: 7023420779426058240 op_type: WRITE_OP write_request { tablet_id: "d846ad6206514c79b142064043ad1cb0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\262YF\016\370s9}\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""749290570484b570" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023420779377184768 authz_token { token_data: "\010\251\222\321\261\006\"3\n\005slave\022*\n 46a3ef9b51d44a84a25ae9ff7198c9b5\020\001\030\001 \001(\001" signature: "<redacted>""/\254!D\336[\307.g\r1\374C\000.\363M\244\212\322\025*)\'\222\213!AL\253N\233?\367s~{\332\006t\023\201\036X+? \327\013\375\033\034\264\347[\344\277\227\201]wG\252_" signing_key_seq_num: 0 } } request_id { client_id: "4abe53ca85cc4d56b5f487c687512ae1" seq_no: 134 first_incomplete_seq_no: 134 attempt_no: 0 }
I20240503 02:12:18.832638  6162 raft_consensus.cc:2258] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.834081  6162 ts_tablet_manager.cc:1875] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:18.841559  6162 ts_tablet_manager.cc:1888] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.40
I20240503 02:12:18.842360  6162 log.cc:1196] T d846ad6206514c79b142064043ad1cb0 P d7f19d54f56849f495dc547de6a47ad2: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/d846ad6206514c79b142064043ad1cb0
I20240503 02:12:18.843375  6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a2da7ca68b43e28cad0847e4d27982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.846833  3908 catalog_manager.cc:4830] TS d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433): tablet d846ad6206514c79b142064043ad1cb0 (table rebalance_test_table_2 [id=46a3ef9b51d44a84a25ae9ff7198c9b5]) successfully deleted
I20240503 02:12:18.875777  6047 tablet_copy_client.cc:783] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:18.876646  6047 tablet_copy_client.cc:647] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:18.879362  6047 tablet_copy_client.cc:523] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:18.886382  6047 tablet_bootstrap.cc:492] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc: Bootstrap starting.
W20240503 02:12:18.902087  4260 consensus_peers.cc:485] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc -> Peer 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5:39125): Couldn't send request to peer 37c60eb66a6b45aaac4e0d7061572da6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c1ff96ff85b4d5e9bd80fe5838b16db. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:18.937801  4345 tablet_copy_service.cc:140] P 8b812251ceb44fbe8a54fea04a7f8dfc: Received BeginTabletCopySession request for tablet 8334f743021f4957a427800af656bec1 from peer 9c804fe67ee34e7fae5fab23db90f298 ({username='slave'} at 127.0.145.6:48257)
I20240503 02:12:18.938051  4345 tablet_copy_service.cc:161] P 8b812251ceb44fbe8a54fea04a7f8dfc: Beginning new tablet copy session on tablet 8334f743021f4957a427800af656bec1 from peer 9c804fe67ee34e7fae5fab23db90f298 at {username='slave'} at 127.0.145.6:48257: session id = 9c804fe67ee34e7fae5fab23db90f298-8334f743021f4957a427800af656bec1
I20240503 02:12:18.941682  4345 tablet_copy_source_session.cc:205] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:18.943447  6083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8334f743021f4957a427800af656bec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:18.954764  6083 tablet_copy_client.cc:783] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:18.955106  6083 tablet_copy_client.cc:647] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:18.957490  6083 tablet_copy_client.cc:523] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:18.968312  6083 tablet_bootstrap.cc:492] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap starting.
I20240503 02:12:18.988776  5374 raft_consensus.cc:1203] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.61->[1.62-1.87]   Dedup: 1.83->[1.84-1.87]
W20240503 02:12:19.035331  3998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.6:36423, user_credentials={real_user=slave}} blocked reactor thread for 64714.7us
I20240503 02:12:19.036465  4192 consensus_queue.cc:237] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 1.122, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } }
I20240503 02:12:19.064774  5504 raft_consensus.cc:1203] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.83->[1.84-1.95]   Dedup: 1.88->[1.89-1.95]
I20240503 02:12:19.083252  4062 raft_consensus.cc:1261] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240503 02:12:19.089704  6172 consensus_queue.cc:1035] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20240503 02:12:19.107797  5843 raft_consensus.cc:2932] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Committing config change with OpId 1.123: config changed from index -1 to 123, NON_VOTER 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } } }
I20240503 02:12:19.116811  5862 raft_consensus.cc:1050] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc: attempting to promote NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc to VOTER
I20240503 02:12:19.117934  5862 consensus_queue.cc:237] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 1.97, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
W20240503 02:12:19.119112  4130 consensus_peers.cc:485] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 -> Peer 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4:33659): Couldn't send request to peer 6ebeaff495e042c78067c5b18600dc98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d98afd14a7f14415a0f404dea8750d82. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:19.122012  4062 raft_consensus.cc:1261] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240503 02:12:19.125108  5890 raft_consensus.cc:1050] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7: attempting to promote NON_VOTER 6ebeaff495e042c78067c5b18600dc98 to VOTER
I20240503 02:12:19.125941  5890 consensus_queue.cc:237] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } }
I20240503 02:12:19.127537  6124 consensus_queue.cc:1035] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20240503 02:12:19.129688  5765 raft_consensus.cc:1261] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240503 02:12:19.130863  6124 consensus_queue.cc:1035] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20240503 02:12:19.133989  6124 raft_consensus.cc:2932] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.98: config changed from index 89 to 98, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:19.138013  4062 raft_consensus.cc:2932] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index 89 to 98, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:19.144323  5765 raft_consensus.cc:2932] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index 89 to 98, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:19.156049  3918 catalog_manager.cc:5484] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: config changed from index -1 to 123, NON_VOTER 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) added. New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:19.160301  5374 raft_consensus.cc:1261] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240503 02:12:19.162328  3920 catalog_manager.cc:5484] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index 89 to 98, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:19.170202  4322 consensus_queue.cc:237] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:19.170655  6194 consensus_queue.cc:1035] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20240503 02:12:19.174672  5765 raft_consensus.cc:1261] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 02:12:19.175338  6124 consensus_queue.cc:1035] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240503 02:12:19.179535  6194 raft_consensus.cc:2932] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index 62 to 88, 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:19.180202  5374 raft_consensus.cc:2932] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index 62 to 88, 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:19.186543  3919 catalog_manager.cc:5484] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 reported cstate change: config changed from index 62 to 88, 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:19.188699  6147 tablet_bootstrap.cc:492] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98: Bootstrap replayed 1/1 log segments. Stats: ops{read=55 overwritten=0 applied=55 ignored=0} inserts{seen=686 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:19.189190  6147 tablet_bootstrap.cc:492] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98: Bootstrap complete.
I20240503 02:12:19.189508  6147 ts_tablet_manager.cc:1367] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98: Time spent bootstrapping tablet: real 0.481s	user 0.128s	sys 0.038s
I20240503 02:12:19.190649  6147 raft_consensus.cc:348] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } }
I20240503 02:12:19.190897  6147 raft_consensus.cc:729] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ebeaff495e042c78067c5b18600dc98, State: Initialized, Role: LEARNER
I20240503 02:12:19.191112  6147 consensus_queue.cc:260] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 55, Last appended: 1.55, Last appended by leader: 55, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } }
I20240503 02:12:19.191731  6147 ts_tablet_manager.cc:1398] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.004s
I20240503 02:12:19.192499  4345 tablet_copy_service.cc:342] P 8b812251ceb44fbe8a54fea04a7f8dfc: Request end of tablet copy session 6ebeaff495e042c78067c5b18600dc98-ec5c9e506c2f4c29aea9a9de259a0c19 received from {username='slave'} at 127.0.145.4:52377
I20240503 02:12:19.192668  4345 tablet_copy_service.cc:433] P 8b812251ceb44fbe8a54fea04a7f8dfc: ending tablet copy session 6ebeaff495e042c78067c5b18600dc98-ec5c9e506c2f4c29aea9a9de259a0c19 on tablet ec5c9e506c2f4c29aea9a9de259a0c19 with peer 6ebeaff495e042c78067c5b18600dc98
I20240503 02:12:19.195627  6124 raft_consensus.cc:2932] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index 98 to 99, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:19.201143  4192 raft_consensus.cc:1261] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240503 02:12:19.205082  3907 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df156172a4554094b36cd5cc3876c922 with cas_config_opid_index 98: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:19.206261  4061 raft_consensus.cc:2932] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index -1 to 123, NON_VOTER 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } } }
I20240503 02:12:19.208490  5890 consensus_queue.cc:1035] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20240503 02:12:19.211295  4194 raft_consensus.cc:2932] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index 62 to 88, 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:19.212461  5926 raft_consensus.cc:1050] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7: attempting to promote NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 to VOTER
I20240503 02:12:19.213176  5926 consensus_queue.cc:237] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 1.96, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } }
I20240503 02:12:19.215236  4324 raft_consensus.cc:1261] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240503 02:12:19.218209  5924 consensus_queue.cc:1035] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20240503 02:12:19.221633  3915 catalog_manager.cc:5484] T df156172a4554094b36cd5cc3876c922 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index 98 to 99, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:19.227672  6193 raft_consensus.cc:2932] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.97: config changed from index 84 to 97, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:19.230829  5504 raft_consensus.cc:1261] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240503 02:12:19.231961  5924 consensus_queue.cc:1035] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20240503 02:12:19.233904  5504 raft_consensus.cc:2932] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 84 to 97, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:19.236630  4322 raft_consensus.cc:2932] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 84 to 97, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:19.246515  4043 tablet_service.cc:1514] Processing DeleteTablet for tablet df156172a4554094b36cd5cc3876c922 with delete_type TABLET_DATA_TOMBSTONED (TS c4af56cee8e546c384cf20f40deeacd7 not found in new config with opid_index 99) from {username='slave'} at 127.0.0.1:51996
I20240503 02:12:19.270996  3915 catalog_manager.cc:5484] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 reported cstate change: config changed from index 84 to 97, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:19.277746  6083 tablet_bootstrap.cc:492] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap replayed 1/1 log segments. Stats: ops{read=115 overwritten=0 applied=115 ignored=0} inserts{seen=746 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:19.278667  6083 tablet_bootstrap.cc:492] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap complete.
I20240503 02:12:19.278996  6083 ts_tablet_manager.cc:1367] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298: Time spent bootstrapping tablet: real 0.311s	user 0.165s	sys 0.038s
I20240503 02:12:19.280040  6083 raft_consensus.cc:348] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
I20240503 02:12:19.280295  6083 raft_consensus.cc:729] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9c804fe67ee34e7fae5fab23db90f298, State: Initialized, Role: LEARNER
I20240503 02:12:19.280516  6083 consensus_queue.cc:260] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 115, Last appended: 1.115, Last appended by leader: 115, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
I20240503 02:12:19.281121  6083 ts_tablet_manager.cc:1398] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
W20240503 02:12:19.281522  4130 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.1:43835, user_credentials={real_user=slave}} blocked reactor thread for 53995.1us
I20240503 02:12:19.284137  5765 raft_consensus.cc:2932] T df156172a4554094b36cd5cc3876c922 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 98 to 99, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:19.307041  6211 ts_tablet_manager.cc:902] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6: Initiating tablet copy from peer 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:19.307490  4061 consensus_queue.cc:237] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 1.99, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } }
I20240503 02:12:19.308633  6210 tablet_replica.cc:330] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7: stopping tablet replica
I20240503 02:12:19.309652  5505 raft_consensus.cc:1261] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 99. Preceding OpId from leader: term: 1 index: 100. (index mismatch)
I20240503 02:12:19.310070  6210 raft_consensus.cc:2229] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:19.310683  6210 raft_consensus.cc:2258] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.311131  6130 consensus_queue.cc:1035] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 98, Time since last communication: 0.000s
I20240503 02:12:19.312229  4345 tablet_copy_service.cc:342] P 8b812251ceb44fbe8a54fea04a7f8dfc: Request end of tablet copy session 9c804fe67ee34e7fae5fab23db90f298-8334f743021f4957a427800af656bec1 received from {username='slave'} at 127.0.145.6:48257
I20240503 02:12:19.312374  4345 tablet_copy_service.cc:433] P 8b812251ceb44fbe8a54fea04a7f8dfc: ending tablet copy session 9c804fe67ee34e7fae5fab23db90f298-8334f743021f4957a427800af656bec1 on tablet 8334f743021f4957a427800af656bec1 with peer 9c804fe67ee34e7fae5fab23db90f298
I20240503 02:12:19.314105  5924 raft_consensus.cc:2932] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.100: config changed from index 97 to 100, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:19.316058  5374 raft_consensus.cc:1203] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.48->[1.49-1.57]   Dedup: 1.55->[1.56-1.57]
I20240503 02:12:19.317096  6210 ts_tablet_manager.cc:1875] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:19.319574  3909 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8b76c95bac0f497f8ab21c5295028026 with cas_config_opid_index 97: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:19.323243  5505 raft_consensus.cc:2932] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Committing config change with OpId 1.100: config changed from index 97 to 100, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:19.332298  3920 catalog_manager.cc:5484] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 reported cstate change: config changed from index 97 to 100, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:19.336583  4305 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b76c95bac0f497f8ab21c5295028026 with delete_type TABLET_DATA_TOMBSTONED (TS 8b812251ceb44fbe8a54fea04a7f8dfc not found in new config with opid_index 100) from {username='slave'} at 127.0.0.1:54992
I20240503 02:12:19.365413  6218 tablet_replica.cc:330] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc: stopping tablet replica
I20240503 02:12:19.365659  6218 raft_consensus.cc:2229] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:19.365898  6218 pending_rounds.cc:62] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc: Trying to abort 1 pending ops.
I20240503 02:12:19.366017  6218 pending_rounds.cc:68] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc: Aborting op as it isn't in flight: id { term: 1 index: 99 } timestamp: 7023420781725810688 op_type: WRITE_OP write_request { tablet_id: "8b76c95bac0f497f8ab21c5295028026" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\240o\234c\375\232i\224\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""835acbc17beddb3e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023420781485330432 authz_token { token_data: "\010\251\222\321\261\006\"3\n\005slave\022*\n 76e610e6ec844c2da65036194e355bbe\020\001\030\001 \001(\001" signature: "<redacted>""\005h@<\313\370\025\002\005\032\303\024\030g}\024\347\214\205h\352_c\373\005\360*\264%\262\210T\274\nS\3762@\017\354\305\031\371F0\025\007qsk\313\3335\257\212\300%a\301\220\334\373\020\212" signing_key_seq_num: 0 } } request_id { client_id: "6b74c88fe8954ef6b96aa35cbccaeba3" seq_no: 276 first_incomplete_seq_no: 276 attempt_no: 0 }
I20240503 02:12:19.369333  6218 raft_consensus.cc:2258] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.374145  6218 ts_tablet_manager.cc:1875] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:19.374300  4194 consensus_queue.cc:237] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 1.100, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
I20240503 02:12:19.382557  4324 raft_consensus.cc:1261] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 101. (index mismatch)
I20240503 02:12:19.383425  6172 consensus_queue.cc:1035] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20240503 02:12:19.386960  6210 ts_tablet_manager.cc:1888] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.98
I20240503 02:12:19.387427  6210 log.cc:1196] T df156172a4554094b36cd5cc3876c922 P c4af56cee8e546c384cf20f40deeacd7: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/df156172a4554094b36cd5cc3876c922
I20240503 02:12:19.388042  6031 raft_consensus.cc:2932] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Committing config change with OpId 1.101: config changed from index -1 to 101, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } } }
I20240503 02:12:19.392211  3909 catalog_manager.cc:4830] TS c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835): tablet df156172a4554094b36cd5cc3876c922 (table rebalance_test_table_1 [id=908c33982a334c0a957523734a0931d3]) successfully deleted
I20240503 02:12:19.392321  6208 raft_consensus.cc:1050] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc: attempting to promote NON_VOTER 6ebeaff495e042c78067c5b18600dc98 to VOTER
I20240503 02:12:19.393028  6208 consensus_queue.cc:237] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } }
I20240503 02:12:19.394083  4324 raft_consensus.cc:2932] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Committing config change with OpId 1.101: config changed from index -1 to 101, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } } }
I20240503 02:12:19.396196  5373 raft_consensus.cc:1261] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 02:12:19.397760  6207 consensus_queue.cc:1035] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20240503 02:12:19.400764  6038 raft_consensus.cc:2932] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index 49 to 59, 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:19.404511  3919 catalog_manager.cc:5484] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: config changed from index -1 to 101, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:19.404510  5373 raft_consensus.cc:2932] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index 49 to 59, 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:19.417922  6218 ts_tablet_manager.cc:1888] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.99
I20240503 02:12:19.419075  6218 log.cc:1196] T 8b76c95bac0f497f8ab21c5295028026 P 8b812251ceb44fbe8a54fea04a7f8dfc: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/8b76c95bac0f497f8ab21c5295028026
I20240503 02:12:19.421340  3907 catalog_manager.cc:4830] TS 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599): tablet 8b76c95bac0f497f8ab21c5295028026 (table rebalance_test_table_3 [id=76e610e6ec844c2da65036194e355bbe]) successfully deleted
I20240503 02:12:19.432058  3919 catalog_manager.cc:5484] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index 49 to 59, 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:19.439030  4322 consensus_queue.cc:237] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } }
I20240503 02:12:19.468024  5374 raft_consensus.cc:1261] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20240503 02:12:19.469122  6038 consensus_queue.cc:1035] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 59, Time since last communication: 0.000s
I20240503 02:12:19.475970  5374 raft_consensus.cc:2932] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98 [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index 59 to 61, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:19.482123  4194 raft_consensus.cc:1261] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 02:12:19.496034  6211 tablet_copy_client.cc:308] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:41599
W20240503 02:12:19.502897  4129 consensus_peers.cc:485] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 -> Peer 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6:36423): Couldn't send request to peer 9c804fe67ee34e7fae5fab23db90f298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07df29aeb41b427a8dcaf4a1a7db30da. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:19.503976  3919 catalog_manager.cc:5484] T ec5c9e506c2f4c29aea9a9de259a0c19 P 6ebeaff495e042c78067c5b18600dc98 reported cstate change: config changed from index 59 to 61, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:19.515478  5862 raft_consensus.cc:2932] T ec5c9e506c2f4c29aea9a9de259a0c19 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index 59 to 61, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:19.519304  4345 tablet_copy_service.cc:140] P 8b812251ceb44fbe8a54fea04a7f8dfc: Received BeginTabletCopySession request for tablet 1c1ff96ff85b4d5e9bd80fe5838b16db from peer 37c60eb66a6b45aaac4e0d7061572da6 ({username='slave'} at 127.0.145.5:50439)
I20240503 02:12:19.519510  4345 tablet_copy_service.cc:161] P 8b812251ceb44fbe8a54fea04a7f8dfc: Beginning new tablet copy session on tablet 1c1ff96ff85b4d5e9bd80fe5838b16db from peer 37c60eb66a6b45aaac4e0d7061572da6 at {username='slave'} at 127.0.145.5:50439: session id = 37c60eb66a6b45aaac4e0d7061572da6-1c1ff96ff85b4d5e9bd80fe5838b16db
I20240503 02:12:19.523061  4345 tablet_copy_source_session.cc:205] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:19.525770  6211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c1ff96ff85b4d5e9bd80fe5838b16db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.530956  4173 tablet_service.cc:1514] Processing DeleteTablet for tablet ec5c9e506c2f4c29aea9a9de259a0c19 with delete_type TABLET_DATA_TOMBSTONED (TS d7f19d54f56849f495dc547de6a47ad2 not found in new config with opid_index 61) from {username='slave'} at 127.0.0.1:55800
I20240503 02:12:19.532256  3907 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ec5c9e506c2f4c29aea9a9de259a0c19 with cas_config_opid_index 59: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:19.541646  5634 raft_consensus.cc:1203] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.101->[1.102-1.117]   Dedup: 1.115->[1.116-1.117]
I20240503 02:12:19.547463  6211 tablet_copy_client.cc:783] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:19.547801  6211 tablet_copy_client.cc:647] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:19.549742  6234 tablet_replica.cc:330] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2: stopping tablet replica
I20240503 02:12:19.550103  6234 raft_consensus.cc:2229] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:19.550438  6234 raft_consensus.cc:2258] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.551471  6234 ts_tablet_manager.cc:1875] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:19.554721  6211 tablet_copy_client.cc:523] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:19.558995  6234 ts_tablet_manager.cc:1888] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.58
I20240503 02:12:19.559252  6234 log.cc:1196] T ec5c9e506c2f4c29aea9a9de259a0c19 P d7f19d54f56849f495dc547de6a47ad2: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/ec5c9e506c2f4c29aea9a9de259a0c19
I20240503 02:12:19.561471  3908 catalog_manager.cc:4830] TS d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433): tablet ec5c9e506c2f4c29aea9a9de259a0c19 (table rebalance_test_table_2 [id=46a3ef9b51d44a84a25ae9ff7198c9b5]) successfully deleted
I20240503 02:12:19.581808  6211 tablet_bootstrap.cc:492] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6: Bootstrap starting.
I20240503 02:12:19.585855  6147 ts_tablet_manager.cc:902] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98: Initiating tablet copy from peer d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:19.588371  4061 consensus_queue.cc:237] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } }
W20240503 02:12:19.602013  3999 consensus_peers.cc:485] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 -> Peer 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7:35457): Couldn't send request to peer 6c3559ccff5d4c6681b3c90f892143dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 368fca99d20d4d6e826c45e44379c056. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:19.603399  4194 raft_consensus.cc:1261] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240503 02:12:19.611347  6211 tablet_bootstrap.cc:492] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:19.611857  6211 tablet_bootstrap.cc:492] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6: Bootstrap complete.
I20240503 02:12:19.612145  6211 ts_tablet_manager.cc:1367] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6: Time spent bootstrapping tablet: real 0.030s	user 0.007s	sys 0.004s
I20240503 02:12:19.613057  6211 raft_consensus.cc:348] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } }
I20240503 02:12:19.613373  6211 raft_consensus.cc:729] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 37c60eb66a6b45aaac4e0d7061572da6, State: Initialized, Role: LEARNER
I20240503 02:12:19.613649  6211 consensus_queue.cc:260] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } }
I20240503 02:12:19.618863  6211 ts_tablet_manager.cc:1398] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.003s
I20240503 02:12:19.619833  4345 tablet_copy_service.cc:342] P 8b812251ceb44fbe8a54fea04a7f8dfc: Request end of tablet copy session 37c60eb66a6b45aaac4e0d7061572da6-1c1ff96ff85b4d5e9bd80fe5838b16db received from {username='slave'} at 127.0.145.5:50439
I20240503 02:12:19.619972  4345 tablet_copy_service.cc:433] P 8b812251ceb44fbe8a54fea04a7f8dfc: ending tablet copy session 37c60eb66a6b45aaac4e0d7061572da6-1c1ff96ff85b4d5e9bd80fe5838b16db on tablet 1c1ff96ff85b4d5e9bd80fe5838b16db with peer 37c60eb66a6b45aaac4e0d7061572da6
I20240503 02:12:19.624897  6193 consensus_queue.cc:1035] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20240503 02:12:19.628249  6047 tablet_bootstrap.cc:492] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=116 overwritten=0 applied=115 ignored=0} inserts{seen=800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:12:19.628878  6047 tablet_bootstrap.cc:492] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc: Bootstrap complete.
I20240503 02:12:19.629174  6047 ts_tablet_manager.cc:1367] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc: Time spent bootstrapping tablet: real 0.743s	user 0.176s	sys 0.034s
I20240503 02:12:19.630030  6047 raft_consensus.cc:348] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } }
I20240503 02:12:19.632818  6147 tablet_copy_client.cc:308] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:44433
I20240503 02:12:19.638439  6047 raft_consensus.cc:729] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c3559ccff5d4c6681b3c90f892143dc, State: Initialized, Role: LEARNER
I20240503 02:12:19.638865  6047 consensus_queue.cc:260] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 115, Last appended: 1.116, Last appended by leader: 116, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } }
I20240503 02:12:19.639662  6047 ts_tablet_manager.cc:1398] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20240503 02:12:19.640627  4214 tablet_copy_service.cc:342] P d7f19d54f56849f495dc547de6a47ad2: Request end of tablet copy session 6c3559ccff5d4c6681b3c90f892143dc-75a2da7ca68b43e28cad0847e4d27982 received from {username='slave'} at 127.0.145.7:42803
I20240503 02:12:19.640763  4214 tablet_copy_service.cc:433] P d7f19d54f56849f495dc547de6a47ad2: ending tablet copy session 6c3559ccff5d4c6681b3c90f892143dc-75a2da7ca68b43e28cad0847e4d27982 on tablet 75a2da7ca68b43e28cad0847e4d27982 with peer 6c3559ccff5d4c6681b3c90f892143dc
I20240503 02:12:19.642818  5890 raft_consensus.cc:2932] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index -1 to 59, NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } } }
I20240503 02:12:19.648618  5862 raft_consensus.cc:1050] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc: attempting to promote NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 to VOTER
I20240503 02:12:19.649431  5862 consensus_queue.cc:237] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 1.118, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } }
I20240503 02:12:19.650645  4214 tablet_copy_service.cc:140] P d7f19d54f56849f495dc547de6a47ad2: Received BeginTabletCopySession request for tablet d98afd14a7f14415a0f404dea8750d82 from peer 6ebeaff495e042c78067c5b18600dc98 ({username='slave'} at 127.0.145.4:50117)
I20240503 02:12:19.650840  4214 tablet_copy_service.cc:161] P d7f19d54f56849f495dc547de6a47ad2: Beginning new tablet copy session on tablet d98afd14a7f14415a0f404dea8750d82 from peer 6ebeaff495e042c78067c5b18600dc98 at {username='slave'} at 127.0.145.4:50117: session id = 6ebeaff495e042c78067c5b18600dc98-d98afd14a7f14415a0f404dea8750d82
I20240503 02:12:19.651691  5634 raft_consensus.cc:1261] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240503 02:12:19.651547  4192 raft_consensus.cc:2932] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index -1 to 59, NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } } }
I20240503 02:12:19.652454  6207 consensus_queue.cc:1035] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20240503 02:12:19.653614  4062 raft_consensus.cc:1261] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240503 02:12:19.654062  4214 tablet_copy_source_session.cc:205] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:19.654254  6207 consensus_queue.cc:1035] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20240503 02:12:19.656199  6207 raft_consensus.cc:2932] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.119: config changed from index 102 to 119, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:19.657411  6147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d98afd14a7f14415a0f404dea8750d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.660645  5634 raft_consensus.cc:2932] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index 102 to 119, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:19.661917  4062 raft_consensus.cc:2932] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index 102 to 119, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:19.669271  3915 catalog_manager.cc:5484] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: config changed from index -1 to 59, NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } } }
I20240503 02:12:19.675382  3920 catalog_manager.cc:5484] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298 reported cstate change: config changed from index 102 to 119, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:19.687795  4324 consensus_queue.cc:237] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 1.121, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } }
I20240503 02:12:19.689859  5505 raft_consensus.cc:1203] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240503 02:12:19.691164  5635 raft_consensus.cc:1261] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240503 02:12:19.695120  6208 consensus_queue.cc:1035] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
I20240503 02:12:19.703824  6208 raft_consensus.cc:2932] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.122: config changed from index 119 to 122, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:19.707247  6147 tablet_copy_client.cc:783] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:19.708662  5634 raft_consensus.cc:2932] T 8334f743021f4957a427800af656bec1 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 119 to 122, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:19.710049  3907 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8334f743021f4957a427800af656bec1 with cas_config_opid_index 119: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:19.719298  3920 catalog_manager.cc:5484] T 8334f743021f4957a427800af656bec1 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index 119 to 122, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:19.707839  6147 tablet_copy_client.cc:647] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:19.728961  4043 tablet_service.cc:1514] Processing DeleteTablet for tablet 8334f743021f4957a427800af656bec1 with delete_type TABLET_DATA_TOMBSTONED (TS c4af56cee8e546c384cf20f40deeacd7 not found in new config with opid_index 122) from {username='slave'} at 127.0.0.1:51996
I20240503 02:12:19.729477  6210 tablet_replica.cc:330] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7: stopping tablet replica
I20240503 02:12:19.729697  6210 raft_consensus.cc:2229] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:19.729872  6210 pending_rounds.cc:62] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7: Trying to abort 1 pending ops.
I20240503 02:12:19.729998  6210 pending_rounds.cc:68] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7: Aborting op as it isn't in flight: id { term: 1 index: 120 } timestamp: 7023420783251722240 op_type: WRITE_OP write_request { tablet_id: "8334f743021f4957a427800af656bec1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\226\276\216\014wC\232\\\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""01155f01c420b811" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023420783062704128 authz_token { token_data: "\010\251\222\321\261\006\"3\n\005slave\022*\n 3a36c423c3f74ce392ff73837341d077\020\001\030\001 \001(\001" signature: "<redacted>""u\310\240\375v\372\030J\224_\300\223!\202Z\304\005\373\3314%\271S\336\022(F\244\263#\241\376M\014,\2018\331t\255k~\004\330\024\013\372\236\361\364\310\307\023r\230\305\031q\273rD^+\357" signing_key_seq_num: 0 } } request_id { client_id: "4bd5337196d1463abc1d85e20605f124" seq_no: 356 first_incomplete_seq_no: 356 attempt_no: 0 }
I20240503 02:12:19.730767  6210 raft_consensus.cc:2258] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:19.731734  6147 tablet_copy_client.cc:523] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:19.733469  6210 ts_tablet_manager.cc:1875] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:19.743678  6147 tablet_bootstrap.cc:492] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98: Bootstrap starting.
I20240503 02:12:19.745321  4322 consensus_queue.cc:237] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } }
I20240503 02:12:19.745605  6210 ts_tablet_manager.cc:1888] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.120
I20240503 02:12:19.745757  6210 log.cc:1196] T 8334f743021f4957a427800af656bec1 P c4af56cee8e546c384cf20f40deeacd7: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/8334f743021f4957a427800af656bec1
I20240503 02:12:19.746650  3909 catalog_manager.cc:4830] TS c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835): tablet 8334f743021f4957a427800af656bec1 (table rebalance_test_table_0 [id=3a36c423c3f74ce392ff73837341d077]) successfully deleted
I20240503 02:12:19.750348  4192 raft_consensus.cc:1261] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
W20240503 02:12:19.750675  4260 consensus_peers.cc:485] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc -> Peer 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5:39125): Couldn't send request to peer 37c60eb66a6b45aaac4e0d7061572da6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27609e0e32a34efc915bbff6750f8234. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:19.751233  6208 consensus_queue.cc:1035] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240503 02:12:19.755137  6207 raft_consensus.cc:2932] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index -1 to 82, NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } } }
I20240503 02:12:19.760160  4192 raft_consensus.cc:2932] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index -1 to 82, NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } } }
I20240503 02:12:19.764225  3915 catalog_manager.cc:5484] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index -1 to 82, NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:19.902456  4062 consensus_queue.cc:237] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.102, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
W20240503 02:12:19.906984  3998 consensus_peers.cc:485] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 -> Peer 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6:36423): Couldn't send request to peer 9c804fe67ee34e7fae5fab23db90f298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccccc26d94514b87a91b0c00fa18e669. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:19.912174  4324 raft_consensus.cc:1261] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240503 02:12:19.919327  5924 consensus_queue.cc:1035] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240503 02:12:19.925591  5924 raft_consensus.cc:2932] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.103: config changed from index -1 to 103, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } } }
I20240503 02:12:19.931706  3920 catalog_manager.cc:5484] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index -1 to 103, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:19.957262  4324 raft_consensus.cc:2932] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index -1 to 103, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } } }
I20240503 02:12:20.005296  5764 raft_consensus.cc:1203] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.109->[1.110-1.126]   Dedup: 1.116->[1.117-1.126]
I20240503 02:12:20.036559  6047 ts_tablet_manager.cc:902] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc: Initiating tablet copy from peer c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:20.141166  6258 ts_tablet_manager.cc:902] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298: Initiating tablet copy from peer d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:20.182657  6207 raft_consensus.cc:1050] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc: attempting to promote NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 to VOTER
I20240503 02:12:20.183532  6207 consensus_queue.cc:237] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } }
I20240503 02:12:20.188073  6147 tablet_bootstrap.cc:492] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98: Bootstrap replayed 1/1 log segments. Stats: ops{read=125 overwritten=0 applied=125 ignored=0} inserts{seen=797 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:20.188560  6147 tablet_bootstrap.cc:492] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98: Bootstrap complete.
I20240503 02:12:20.188861  6147 ts_tablet_manager.cc:1367] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98: Time spent bootstrapping tablet: real 0.446s	user 0.194s	sys 0.010s
I20240503 02:12:20.189738  6147 raft_consensus.cc:348] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } }
I20240503 02:12:20.189967  6147 raft_consensus.cc:729] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ebeaff495e042c78067c5b18600dc98, State: Initialized, Role: LEARNER
I20240503 02:12:20.190196  6147 consensus_queue.cc:260] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 125, Last appended: 1.125, Last appended by leader: 125, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } }
I20240503 02:12:20.259213  6047 tablet_copy_client.cc:308] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:43835
I20240503 02:12:20.260452  4083 tablet_copy_service.cc:140] P c4af56cee8e546c384cf20f40deeacd7: Received BeginTabletCopySession request for tablet 368fca99d20d4d6e826c45e44379c056 from peer 6c3559ccff5d4c6681b3c90f892143dc ({username='slave'} at 127.0.145.7:41467)
I20240503 02:12:20.260656  4083 tablet_copy_service.cc:161] P c4af56cee8e546c384cf20f40deeacd7: Beginning new tablet copy session on tablet 368fca99d20d4d6e826c45e44379c056 from peer 6c3559ccff5d4c6681b3c90f892143dc at {username='slave'} at 127.0.145.7:41467: session id = 6c3559ccff5d4c6681b3c90f892143dc-368fca99d20d4d6e826c45e44379c056
I20240503 02:12:20.264176  4083 tablet_copy_source_session.cc:205] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:20.267796  6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368fca99d20d4d6e826c45e44379c056. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.290647  6047 tablet_copy_client.cc:783] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:20.291083  6047 tablet_copy_client.cc:647] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:20.298426  6047 tablet_copy_client.cc:523] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:20.306869  6047 tablet_bootstrap.cc:492] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc: Bootstrap starting.
I20240503 02:12:20.438830  6288 ts_tablet_manager.cc:902] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298: Initiating tablet copy from peer c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:20.447595  6271 ts_tablet_manager.cc:902] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6: Initiating tablet copy from peer 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:20.449502  5843 raft_consensus.cc:1050] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2: attempting to promote NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc to VOTER
I20240503 02:12:20.450301  5843 consensus_queue.cc:237] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 132, Committed index: 132, Last appended: 1.133, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:20.460412  5763 raft_consensus.cc:1261] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 134. (index mismatch)
I20240503 02:12:20.461709  5843 consensus_queue.cc:1035] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 132, Time since last communication: 0.000s
I20240503 02:12:20.464674  6288 tablet_copy_client.cc:308] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:43835
I20240503 02:12:20.466382  6166 raft_consensus.cc:2932] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Committing config change with OpId 1.134: config changed from index 110 to 134, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:20.472035  5763 raft_consensus.cc:2932] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 FOLLOWER]: Committing config change with OpId 1.134: config changed from index 110 to 134, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:20.479923  3920 catalog_manager.cc:5484] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc reported cstate change: config changed from index 110 to 134, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:20.493252  4192 raft_consensus.cc:1261] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:20.493237  5503 raft_consensus.cc:1261] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:20.494357  5862 consensus_queue.cc:1035] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:20.495165  5862 consensus_queue.cc:1035] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:20.499500  6038 raft_consensus.cc:2932] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:20.505831  4192 raft_consensus.cc:2932] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:20.510757  3919 catalog_manager.cc:5484] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index 2 to 3, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:20.512939  3997 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.3:41599, user_credentials={real_user=slave}} blocked reactor thread for 69004.1us
I20240503 02:12:20.514410  4083 tablet_copy_service.cc:140] P c4af56cee8e546c384cf20f40deeacd7: Received BeginTabletCopySession request for tablet ccccc26d94514b87a91b0c00fa18e669 from peer 9c804fe67ee34e7fae5fab23db90f298 ({username='slave'} at 127.0.145.6:39277)
I20240503 02:12:20.514748  4083 tablet_copy_service.cc:161] P c4af56cee8e546c384cf20f40deeacd7: Beginning new tablet copy session on tablet ccccc26d94514b87a91b0c00fa18e669 from peer 9c804fe67ee34e7fae5fab23db90f298 at {username='slave'} at 127.0.145.6:39277: session id = 9c804fe67ee34e7fae5fab23db90f298-ccccc26d94514b87a91b0c00fa18e669
I20240503 02:12:20.518301  4083 tablet_copy_source_session.cc:205] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:20.522083  6288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccccc26d94514b87a91b0c00fa18e669. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.530512  6288 tablet_copy_client.cc:783] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:20.530840  6288 tablet_copy_client.cc:647] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:20.546599  6288 tablet_copy_client.cc:523] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 02:12:20.554219  3998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.6:36423, user_credentials={real_user=slave}} blocked reactor thread for 111442us
I20240503 02:12:20.561554  5503 raft_consensus.cc:2932] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:20.569923  4062 raft_consensus.cc:1261] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 132. Preceding OpId from leader: term: 1 index: 134. (index mismatch)
I20240503 02:12:20.587613  6258 tablet_copy_client.cc:308] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:44433
I20240503 02:12:20.605306  5843 consensus_queue.cc:1035] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 132, Time since last communication: 0.000s
I20240503 02:12:20.652897  6047 tablet_bootstrap.cc:492] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=70 ignored=0} inserts{seen=755 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:20.653401  6047 tablet_bootstrap.cc:492] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc: Bootstrap complete.
I20240503 02:12:20.653774  6047 ts_tablet_manager.cc:1367] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc: Time spent bootstrapping tablet: real 0.347s	user 0.160s	sys 0.003s
I20240503 02:12:20.654908  6047 raft_consensus.cc:348] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } }
I20240503 02:12:20.655217  6047 raft_consensus.cc:729] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c3559ccff5d4c6681b3c90f892143dc, State: Initialized, Role: LEARNER
I20240503 02:12:20.655858  6047 consensus_queue.cc:260] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 70, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: true } }
I20240503 02:12:20.659762  6288 tablet_bootstrap.cc:492] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap starting.
I20240503 02:12:20.668855  6047 ts_tablet_manager.cc:1398] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.000s
W20240503 02:12:20.667083  5296 kernel_stack_watchdog.cc:198] Thread 6147 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 327ms:
Kernel stack:
(could not read kernel stack)
User stack:
    @     0x7f6e70b1a980  (unknown) at ??:0
    @           0x90ef77  __sanitizer::internal_sched_yield() at ??:?
    @           0x87462d  __interceptor_pthread_create at ??:?
    @     0x7f6e6d788d11  kudu::Thread::StartThread() at ??:0
    @           0xa4d187  kudu::Thread::Create() at ??:?
    @     0x7f6e6d7a97ab  kudu::ThreadPool::CreateThread() at ??:0
    @     0x7f6e6d7a53ad  kudu::ThreadPool::DoSubmit() at ??:0
    @     0x7f6e6d7a384f  kudu::ThreadPoolToken::Submit() at ??:0
    @     0x7f6e78f9d4f0  kudu::consensus::RaftConsensus::MarkDirty() at ??:0
    @     0x7f6e78f961b0  kudu::consensus::RaftConsensus::Start() at ??:0
    @     0x7f6e7a2ab9ce  kudu::tablet::TabletReplica::Start() at ??:0
    @     0x7f6e7cd3d554  kudu::tserver::TSTabletManager::OpenTablet() at ??:0
    @     0x7f6e7cd38a29  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
    @     0x7f6e7cd70a8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
    @     0x7f6e7cd53981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
    @     0x7f6e7cd5370e  std::_Function_handler<>::_M_invoke() at ??:0
I20240503 02:12:20.673784  4083 tablet_copy_service.cc:342] P c4af56cee8e546c384cf20f40deeacd7: Request end of tablet copy session 6c3559ccff5d4c6681b3c90f892143dc-368fca99d20d4d6e826c45e44379c056 received from {username='slave'} at 127.0.145.7:41467
I20240503 02:12:20.673961  4083 tablet_copy_service.cc:433] P c4af56cee8e546c384cf20f40deeacd7: ending tablet copy session 6c3559ccff5d4c6681b3c90f892143dc-368fca99d20d4d6e826c45e44379c056 on tablet 368fca99d20d4d6e826c45e44379c056 with peer 6c3559ccff5d4c6681b3c90f892143dc
I20240503 02:12:20.724606  4062 raft_consensus.cc:2932] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Committing config change with OpId 1.134: config changed from index 110 to 134, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:20.765465  6271 tablet_copy_client.cc:308] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:41599
I20240503 02:12:20.766314  4345 tablet_copy_service.cc:140] P 8b812251ceb44fbe8a54fea04a7f8dfc: Received BeginTabletCopySession request for tablet 27609e0e32a34efc915bbff6750f8234 from peer 37c60eb66a6b45aaac4e0d7061572da6 ({username='slave'} at 127.0.145.5:50439)
I20240503 02:12:20.766491  4345 tablet_copy_service.cc:161] P 8b812251ceb44fbe8a54fea04a7f8dfc: Beginning new tablet copy session on tablet 27609e0e32a34efc915bbff6750f8234 from peer 37c60eb66a6b45aaac4e0d7061572da6 at {username='slave'} at 127.0.145.5:50439: session id = 37c60eb66a6b45aaac4e0d7061572da6-27609e0e32a34efc915bbff6750f8234
I20240503 02:12:20.770512  4345 tablet_copy_source_session.cc:205] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:20.773428  6271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27609e0e32a34efc915bbff6750f8234. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:12:20.800819  6147 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.122s	sys 0.300s
W20240503 02:12:20.800977  6147 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.122s	sys 0.300s
I20240503 02:12:20.801291  6147 ts_tablet_manager.cc:1398] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98: Time spent starting tablet: real 0.612s	user 0.122s	sys 0.300s
W20240503 02:12:20.801390  6147 ts_tablet_manager.cc:1430] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98: Tablet startup took 612ms
W20240503 02:12:20.801441  6147 ts_tablet_manager.cc:1432] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98: Trace:
0503 02:12:19.585513 (+     0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.585578 (+    65us) service_pool.cc:225] Handling call
0503 02:12:19.585953 (+   375us) ts_tablet_manager.cc:903] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98: Initiating tablet copy from peer d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
0503 02:12:19.656876 (+ 70923us) tablet_copy_client.cc:353] Tablet copy session begun
0503 02:12:19.663384 (+  6508us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:19.663513 (+   129us) tablet_copy_client.cc:482] Wrote new tablet metadata
0503 02:12:19.702520 (+ 39007us) tablet_copy_client.cc:403] Wrote new consensus metadata
0503 02:12:19.702689 (+   169us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:19.702806 (+   117us) ts_tablet_manager.cc:958] Replica registered.
0503 02:12:19.702831 (+    25us) inbound_call.cc:173] Queueing success response
0503 02:12:19.738847 (+ 36016us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0503 02:12:20.188949 (+450102us) ts_tablet_manager.cc:1399] Starting tablet replica
0503 02:12:20.189147 (+   198us) tablet_replica.cc:259] Starting instrumentation
0503 02:12:20.189296 (+   149us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":85,"raft.queue_time_us":610244,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"tablet-copy.queue_time_us":48,"tablet-download-pool-d98afd14a7f14415a0f404dea8750d82.queue_time_us":50,"tablet-download-pool-d98afd14a7f14415a0f404dea8750d82.run_cpu_time_us":898,"tablet-download-pool-d98afd14a7f14415a0f404dea8750d82.run_wall_time_us":3457,"thread_start_us":656964,"threads_started":5}
I20240503 02:12:20.806568  4214 tablet_copy_service.cc:342] P d7f19d54f56849f495dc547de6a47ad2: Request end of tablet copy session 6ebeaff495e042c78067c5b18600dc98-d98afd14a7f14415a0f404dea8750d82 received from {username='slave'} at 127.0.145.4:50117
I20240503 02:12:20.806802  4214 tablet_copy_service.cc:433] P d7f19d54f56849f495dc547de6a47ad2: ending tablet copy session 6ebeaff495e042c78067c5b18600dc98-d98afd14a7f14415a0f404dea8750d82 on tablet d98afd14a7f14415a0f404dea8750d82 with peer 6ebeaff495e042c78067c5b18600dc98
I20240503 02:12:20.812316  4214 tablet_copy_service.cc:140] P d7f19d54f56849f495dc547de6a47ad2: Received BeginTabletCopySession request for tablet 07df29aeb41b427a8dcaf4a1a7db30da from peer 9c804fe67ee34e7fae5fab23db90f298 ({username='slave'} at 127.0.145.6:50551)
I20240503 02:12:20.812525  4214 tablet_copy_service.cc:161] P d7f19d54f56849f495dc547de6a47ad2: Beginning new tablet copy session on tablet 07df29aeb41b427a8dcaf4a1a7db30da from peer 9c804fe67ee34e7fae5fab23db90f298 at {username='slave'} at 127.0.145.6:50551: session id = 9c804fe67ee34e7fae5fab23db90f298-07df29aeb41b427a8dcaf4a1a7db30da
I20240503 02:12:20.837541  5374 raft_consensus.cc:1203] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.122->[1.123-1.140]   Dedup: 1.125->[1.126-1.140]
I20240503 02:12:20.842113  4214 tablet_copy_source_session.cc:205] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 02:12:20.843901  3998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.6:36423, user_credentials={real_user=slave}} blocked reactor thread for 51111.3us
I20240503 02:12:20.864862  6258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07df29aeb41b427a8dcaf4a1a7db30da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.898789  6271 tablet_copy_client.cc:783] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:20.899130  6271 tablet_copy_client.cc:647] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:20.925343  5843 raft_consensus.cc:1050] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2: attempting to promote NON_VOTER 6ebeaff495e042c78067c5b18600dc98 to VOTER
I20240503 02:12:20.926025  5843 consensus_queue.cc:237] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 140, Committed index: 140, Last appended: 1.141, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 142 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } }
I20240503 02:12:20.928560  5373 raft_consensus.cc:1261] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 141. Preceding OpId from leader: term: 1 index: 142. (index mismatch)
I20240503 02:12:20.929373  6166 consensus_queue.cc:1035] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 142, Last known committed idx: 140, Time since last communication: 0.000s
I20240503 02:12:20.935137  6031 raft_consensus.cc:2932] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Committing config change with OpId 1.142: config changed from index 123 to 142, 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 142 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:20.936343  5373 raft_consensus.cc:2932] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98 [term 1 FOLLOWER]: Committing config change with OpId 1.142: config changed from index 123 to 142, 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 142 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:20.942859  3920 catalog_manager.cc:5484] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98 reported cstate change: config changed from index 123 to 142, 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 142 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:20.962488  6271 tablet_copy_client.cc:523] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:20.966013  4192 consensus_queue.cc:237] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 142, Committed index: 142, Last appended: 1.143, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } }
I20240503 02:12:20.968310  6271 tablet_bootstrap.cc:492] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6: Bootstrap starting.
I20240503 02:12:20.970106  5374 raft_consensus.cc:1261] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98 [term 1 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 143. Preceding OpId from leader: term: 1 index: 144. (index mismatch)
I20240503 02:12:20.974277  6031 consensus_queue.cc:1035] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 144, Last known committed idx: 142, Time since last communication: 0.000s
I20240503 02:12:20.977031  5843 raft_consensus.cc:2932] T d98afd14a7f14415a0f404dea8750d82 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Committing config change with OpId 1.144: config changed from index 142 to 144, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New config: { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:20.982455  3908 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d98afd14a7f14415a0f404dea8750d82 with cas_config_opid_index 142: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:20.983649  5373 raft_consensus.cc:2932] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98 [term 1 FOLLOWER]: Committing config change with OpId 1.144: config changed from index 142 to 144, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New config: { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:20.987612  4061 raft_consensus.cc:1261] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 141. Preceding OpId from leader: term: 1 index: 142. (index mismatch)
I20240503 02:12:20.988546  6258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.145.2:52199 (request call id 1) took 1012 ms. Trace:
I20240503 02:12:20.988711  6258 rpcz_store.cc:276] 0503 02:12:19.976108 (+     0us) service_pool.cc:168] Inserting onto call queue
0503 02:12:19.976171 (+    63us) service_pool.cc:225] Handling call
0503 02:12:20.141275 (+165104us) ts_tablet_manager.cc:903] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298: Initiating tablet copy from peer d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
0503 02:12:20.864325 (+723050us) tablet_copy_client.cc:353] Tablet copy session begun
0503 02:12:20.892255 (+ 27930us) tablet_metadata.cc:641] Metadata flushed
0503 02:12:20.892468 (+   213us) tablet_copy_client.cc:482] Wrote new tablet metadata
0503 02:12:20.988213 (+ 95745us) tablet_copy_client.cc:403] Wrote new consensus metadata
0503 02:12:20.988345 (+   132us) tablet_replica.cc:205] Creating consensus instance
0503 02:12:20.988475 (+   130us) ts_tablet_manager.cc:958] Replica registered.
0503 02:12:20.988503 (+    28us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":135,"tablet-copy.queue_time_us":164745,"thread_start_us":611203,"threads_started":5}
I20240503 02:12:20.996040  3918 catalog_manager.cc:5484] T d98afd14a7f14415a0f404dea8750d82 P 6ebeaff495e042c78067c5b18600dc98 reported cstate change: config changed from index 142 to 144, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:21.000478  4043 tablet_service.cc:1514] Processing DeleteTablet for tablet d98afd14a7f14415a0f404dea8750d82 with delete_type TABLET_DATA_TOMBSTONED (TS c4af56cee8e546c384cf20f40deeacd7 not found in new config with opid_index 144) from {username='slave'} at 127.0.0.1:51996
I20240503 02:12:21.021674  6317 tablet_replica.cc:330] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7: stopping tablet replica
I20240503 02:12:21.021926  6317 raft_consensus.cc:2229] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.022119  6317 pending_rounds.cc:62] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7: Trying to abort 1 pending ops.
I20240503 02:12:21.023340  6317 pending_rounds.cc:68] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7: Aborting op as it isn't in flight: id { term: 1 index: 141 } timestamp: 7023420788393795584 op_type: WRITE_OP write_request { tablet_id: "d98afd14a7f14415a0f404dea8750d82" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\232\002\365\032W\025\206\263\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""57c865bbcb1e0a06" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023420787956924416 authz_token { token_data: "\010\251\222\321\261\006\"3\n\005slave\022*\n 3a36c423c3f74ce392ff73837341d077\020\001\030\001 \001(\001" signature: "<redacted>""u\310\240\375v\372\030J\224_\300\223!\202Z\304\005\373\3314%\271S\336\022(F\244\263#\241\376M\014,\2018\331t\255k~\004\330\024\013\372\236\361\364\310\307\023r\230\305\031q\273rD^+\357" signing_key_seq_num: 0 } } request_id { client_id: "4bd5337196d1463abc1d85e20605f124" seq_no: 397 first_incomplete_seq_no: 397 attempt_no: 0 }
I20240503 02:12:21.023959  6258 tablet_copy_client.cc:783] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:21.024250  6258 tablet_copy_client.cc:647] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:21.026726  6317 raft_consensus.cc:2258] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.026876  6258 tablet_copy_client.cc:523] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:21.031666  6317 ts_tablet_manager.cc:1875] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:21.045826  6258 tablet_bootstrap.cc:492] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap starting.
I20240503 02:12:21.053619  6317 ts_tablet_manager.cc:1888] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.141
I20240503 02:12:21.053781  6317 log.cc:1196] T d98afd14a7f14415a0f404dea8750d82 P c4af56cee8e546c384cf20f40deeacd7: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/d98afd14a7f14415a0f404dea8750d82
I20240503 02:12:21.055291  3909 catalog_manager.cc:4830] TS c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835): tablet d98afd14a7f14415a0f404dea8750d82 (table rebalance_test_table_0 [id=3a36c423c3f74ce392ff73837341d077]) successfully deleted
I20240503 02:12:21.087077  5763 raft_consensus.cc:1203] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.58->[1.59-1.75]   Dedup: 1.70->[1.71-1.75]
I20240503 02:12:21.225114  6194 raft_consensus.cc:1050] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7: attempting to promote NON_VOTER 6c3559ccff5d4c6681b3c90f892143dc to VOTER
I20240503 02:12:21.225919  6194 consensus_queue.cc:237] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:21.229071  5765 raft_consensus.cc:1261] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 LEARNER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240503 02:12:21.229643  5890 consensus_queue.cc:1035] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240503 02:12:21.234259  6193 raft_consensus.cc:2932] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index 59 to 77, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:21.235005  5764 raft_consensus.cc:2932] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 59 to 77, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:21.236711  4194 raft_consensus.cc:1261] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240503 02:12:21.240698  6130 consensus_queue.cc:1035] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240503 02:12:21.241807  3918 catalog_manager.cc:5484] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index 59 to 77, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:21.252717  4194 raft_consensus.cc:2932] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 59 to 77, 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:21.561895  4325 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c1ff96ff85b4d5e9bd80fe5838b16db"
dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:42868
I20240503 02:12:21.562126  4325 raft_consensus.cc:595] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:21.571048  6207 raft_consensus.cc:979] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc: : Instructing follower 37c60eb66a6b45aaac4e0d7061572da6 to start an election
I20240503 02:12:21.571260  6207 raft_consensus.cc:1067] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Signalling peer 37c60eb66a6b45aaac4e0d7061572da6 to start an election
I20240503 02:12:21.577524  5505 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c1ff96ff85b4d5e9bd80fe5838b16db"
dest_uuid: "37c60eb66a6b45aaac4e0d7061572da6"
 from {username='slave'} at 127.0.145.3:52501
I20240503 02:12:21.580513  5505 raft_consensus.cc:482] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:21.581202  5505 raft_consensus.cc:3037] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:21.589118  5505 raft_consensus.cc:504] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } }
I20240503 02:12:21.593665  4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c1ff96ff85b4d5e9bd80fe5838b16db" candidate_uuid: "37c60eb66a6b45aaac4e0d7061572da6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc"
I20240503 02:12:21.593901  4325 raft_consensus.cc:3032] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:21.593914  4192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c1ff96ff85b4d5e9bd80fe5838b16db" candidate_uuid: "37c60eb66a6b45aaac4e0d7061572da6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
I20240503 02:12:21.594039  4325 raft_consensus.cc:729] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Running, Role: LEADER
I20240503 02:12:21.594158  4192 raft_consensus.cc:3037] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:21.594362  4325 consensus_queue.cc:260] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } }
I20240503 02:12:21.594843  4325 raft_consensus.cc:3037] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:21.597677  4192 raft_consensus.cc:2454] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37c60eb66a6b45aaac4e0d7061572da6 in term 2.
I20240503 02:12:21.598433  5440 leader_election.cc:302] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37c60eb66a6b45aaac4e0d7061572da6, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:21.608572  4325 raft_consensus.cc:2454] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37c60eb66a6b45aaac4e0d7061572da6 in term 2.
I20240503 02:12:21.610318  5505 leader_election.cc:288] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [CANDIDATE]: Term 2 election: Requested vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433), 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:21.635231  6340 raft_consensus.cc:2781] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:21.694871  6340 raft_consensus.cc:686] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 LEADER]: Becoming Leader. State: Replica: 37c60eb66a6b45aaac4e0d7061572da6, State: Running, Role: LEADER
I20240503 02:12:21.695575  6340 consensus_queue.cc:237] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } }
I20240503 02:12:21.728802  6271 tablet_bootstrap.cc:492] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6: Bootstrap replayed 1/1 log segments. Stats: ops{read=90 overwritten=0 applied=90 ignored=0} inserts{seen=721 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:21.729271  6271 tablet_bootstrap.cc:492] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6: Bootstrap complete.
I20240503 02:12:21.729548  6271 ts_tablet_manager.cc:1367] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6: Time spent bootstrapping tablet: real 0.761s	user 0.146s	sys 0.026s
I20240503 02:12:21.729512  3918 catalog_manager.cc:5484] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 reported cstate change: term changed from 1 to 2, leader changed from 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) to 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5). New cstate: current_term: 2 leader_uuid: "37c60eb66a6b45aaac4e0d7061572da6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:21.730316  6271 raft_consensus.cc:348] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } }
I20240503 02:12:21.730558  6271 raft_consensus.cc:729] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 37c60eb66a6b45aaac4e0d7061572da6, State: Initialized, Role: LEARNER
I20240503 02:12:21.742328  6271 consensus_queue.cc:260] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 90, Last appended: 1.90, Last appended by leader: 90, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: true } }
I20240503 02:12:21.745427  6271 ts_tablet_manager.cc:1398] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6: Time spent starting tablet: real 0.016s	user 0.001s	sys 0.003s
I20240503 02:12:21.746217  4345 tablet_copy_service.cc:342] P 8b812251ceb44fbe8a54fea04a7f8dfc: Request end of tablet copy session 37c60eb66a6b45aaac4e0d7061572da6-27609e0e32a34efc915bbff6750f8234 received from {username='slave'} at 127.0.145.5:50439
I20240503 02:12:21.746377  4345 tablet_copy_service.cc:433] P 8b812251ceb44fbe8a54fea04a7f8dfc: ending tablet copy session 37c60eb66a6b45aaac4e0d7061572da6-27609e0e32a34efc915bbff6750f8234 on tablet 27609e0e32a34efc915bbff6750f8234 with peer 37c60eb66a6b45aaac4e0d7061572da6
I20240503 02:12:21.917104  6288 tablet_bootstrap.cc:492] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap replayed 1/1 log segments. Stats: ops{read=110 overwritten=0 applied=109 ignored=0} inserts{seen=794 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:12:21.917609  6288 tablet_bootstrap.cc:492] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap complete.
I20240503 02:12:21.917893  6288 ts_tablet_manager.cc:1367] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298: Time spent bootstrapping tablet: real 1.258s	user 0.173s	sys 0.020s
I20240503 02:12:21.923437  6288 raft_consensus.cc:348] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
I20240503 02:12:21.924152  6288 raft_consensus.cc:729] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9c804fe67ee34e7fae5fab23db90f298, State: Initialized, Role: LEARNER
I20240503 02:12:21.924376  6288 consensus_queue.cc:260] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 109, Last appended: 1.110, Last appended by leader: 110, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
I20240503 02:12:21.937855  6288 ts_tablet_manager.cc:1398] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.000s
I20240503 02:12:21.940301  4083 tablet_copy_service.cc:342] P c4af56cee8e546c384cf20f40deeacd7: Request end of tablet copy session 9c804fe67ee34e7fae5fab23db90f298-ccccc26d94514b87a91b0c00fa18e669 received from {username='slave'} at 127.0.145.6:39277
I20240503 02:12:21.940425  4083 tablet_copy_service.cc:433] P c4af56cee8e546c384cf20f40deeacd7: ending tablet copy session 9c804fe67ee34e7fae5fab23db90f298-ccccc26d94514b87a91b0c00fa18e669 on tablet ccccc26d94514b87a91b0c00fa18e669 with peer 9c804fe67ee34e7fae5fab23db90f298
I20240503 02:12:22.112718  4325 raft_consensus.cc:1261] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 2 FOLLOWER]: Refusing update from remote peer 37c60eb66a6b45aaac4e0d7061572da6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:12:22.113462  6340 consensus_queue.cc:1035] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:22.151909  4194 raft_consensus.cc:1261] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Refusing update from remote peer 37c60eb66a6b45aaac4e0d7061572da6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:12:22.152586  6340 consensus_queue.cc:1035] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:22.225152  6258 tablet_bootstrap.cc:492] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap replayed 1/1 log segments. Stats: ops{read=120 overwritten=0 applied=120 ignored=0} inserts{seen=751 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:22.225868  6258 tablet_bootstrap.cc:492] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap complete.
I20240503 02:12:22.227420  6258 ts_tablet_manager.cc:1367] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298: Time spent bootstrapping tablet: real 1.182s	user 0.209s	sys 0.028s
I20240503 02:12:22.231426  6258 raft_consensus.cc:348] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
I20240503 02:12:22.231981  6258 raft_consensus.cc:729] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9c804fe67ee34e7fae5fab23db90f298, State: Initialized, Role: LEARNER
I20240503 02:12:22.232327  6258 consensus_queue.cc:260] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 120, Last appended: 1.120, Last appended by leader: 120, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
I20240503 02:12:22.235039  5505 raft_consensus.cc:1203] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.81->[1.82-1.104]   Dedup: 1.90->[1.91-1.104]
I20240503 02:12:22.235385  6258 ts_tablet_manager.cc:1398] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.000s
I20240503 02:12:22.239439  4214 tablet_copy_service.cc:342] P d7f19d54f56849f495dc547de6a47ad2: Request end of tablet copy session 9c804fe67ee34e7fae5fab23db90f298-07df29aeb41b427a8dcaf4a1a7db30da received from {username='slave'} at 127.0.145.6:50551
I20240503 02:12:22.239590  4214 tablet_copy_service.cc:433] P d7f19d54f56849f495dc547de6a47ad2: ending tablet copy session 9c804fe67ee34e7fae5fab23db90f298-07df29aeb41b427a8dcaf4a1a7db30da on tablet 07df29aeb41b427a8dcaf4a1a7db30da with peer 9c804fe67ee34e7fae5fab23db90f298
I20240503 02:12:22.301532  5634 raft_consensus.cc:1203] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.102->[1.103-1.137]   Dedup: 1.110->[1.111-1.137]
I20240503 02:12:22.465121  5635 raft_consensus.cc:1203] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.100->[1.101-1.137]   Dedup: 1.120->[1.121-1.137]
I20240503 02:12:22.504829  6363 raft_consensus.cc:1050] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc: attempting to promote NON_VOTER 37c60eb66a6b45aaac4e0d7061572da6 to VOTER
I20240503 02:12:22.505412  6031 raft_consensus.cc:1050] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2: attempting to promote NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 to VOTER
I20240503 02:12:22.506024  6363 consensus_queue.cc:237] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } }
I20240503 02:12:22.506294  6031 consensus_queue.cc:237] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 138, Committed index: 138, Last appended: 1.138, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } }
I20240503 02:12:22.508881  5633 raft_consensus.cc:1261] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240503 02:12:22.509516  6166 consensus_queue.cc:1035] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 138, Time since last communication: 0.000s
I20240503 02:12:22.512849  6166 raft_consensus.cc:2932] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Committing config change with OpId 1.139: config changed from index 101 to 139, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:22.514046  5503 raft_consensus.cc:1261] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 LEARNER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch)
I20240503 02:12:22.515434  4192 raft_consensus.cc:1261] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch)
I20240503 02:12:22.517066  4324 raft_consensus.cc:1261] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240503 02:12:22.521565  3920 catalog_manager.cc:5484] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: config changed from index 101 to 139, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:22.523291  5843 consensus_queue.cc:1035] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 138, Time since last communication: 0.000s
I20240503 02:12:22.530126  5890 raft_consensus.cc:1050] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7: attempting to promote NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 to VOTER
I20240503 02:12:22.529923  5633 raft_consensus.cc:2932] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index 101 to 139, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:22.530781  5890 consensus_queue.cc:237] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 141, Committed index: 141, Last appended: 1.142, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } }
I20240503 02:12:22.555846  6207 consensus_queue.cc:1035] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20240503 02:12:22.557375  6207 consensus_queue.cc:1035] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20240503 02:12:22.579818  6207 raft_consensus.cc:2932] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 82 to 107, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
W20240503 02:12:22.582917  3998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.2:44433, user_credentials={real_user=slave}} blocked reactor thread for 50116.7us
I20240503 02:12:22.584280  5633 raft_consensus.cc:1261] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 LEARNER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 1 index: 143. (index mismatch)
I20240503 02:12:22.584839  6396 consensus_queue.cc:1035] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 141, Time since last communication: 0.000s
I20240503 02:12:22.585600  4192 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "75a2da7ca68b43e28cad0847e4d27982"
dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:52776
I20240503 02:12:22.585932  4192 raft_consensus.cc:595] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:22.585654  4324 raft_consensus.cc:2932] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index 101 to 139, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:22.596606  6396 raft_consensus.cc:2932] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.143: config changed from index 103 to 143, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:22.605099  3918 catalog_manager.cc:5484] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index 103 to 143, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:22.603711  5635 raft_consensus.cc:2932] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Committing config change with OpId 1.143: config changed from index 103 to 143, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:22.609858  3915 catalog_manager.cc:5484] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index 82 to 107, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:22.631335  5505 raft_consensus.cc:2932] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 82 to 107, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:22.639165  4194 raft_consensus.cc:2932] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 82 to 107, 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:22.644277  5503 consensus_queue.cc:237] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } }
I20240503 02:12:22.647199  4192 raft_consensus.cc:1261] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Refusing update from remote peer 37c60eb66a6b45aaac4e0d7061572da6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:12:22.647856  6380 consensus_queue.cc:1035] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:22.651293  6380 raft_consensus.cc:2932] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:22.658638  4322 raft_consensus.cc:1261] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 1 index: 143. (index mismatch)
W20240503 02:12:22.662688  4153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
I20240503 02:12:22.667727  4194 raft_consensus.cc:2932] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
W20240503 02:12:22.670377  4153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
I20240503 02:12:22.670452  4322 consensus_queue.cc:237] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 1.108, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } }
I20240503 02:12:22.675505  3908 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c1ff96ff85b4d5e9bd80fe5838b16db with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:22.676223  4193 consensus_queue.cc:237] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 139, Committed index: 139, Last appended: 1.139, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } }
W20240503 02:12:22.677485  4153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
I20240503 02:12:22.678578  5635 raft_consensus.cc:1261] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 139. Preceding OpId from leader: term: 1 index: 140. (index mismatch)
I20240503 02:12:22.679649  6031 consensus_queue.cc:1035] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 140, Last known committed idx: 139, Time since last communication: 0.000s
I20240503 02:12:22.684643  5505 raft_consensus.cc:1261] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 108. Preceding OpId from leader: term: 1 index: 109. (index mismatch)
I20240503 02:12:22.686396  4061 consensus_queue.cc:237] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 143, Committed index: 143, Last appended: 1.143, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } }
I20240503 02:12:22.686666  6363 consensus_queue.cc:1035] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20240503 02:12:22.687165  3918 catalog_manager.cc:5484] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 reported cstate change: config changed from index 3 to 5, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New cstate: current_term: 2 leader_uuid: "37c60eb66a6b45aaac4e0d7061572da6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:22.687950  6031 raft_consensus.cc:2932] T 07df29aeb41b427a8dcaf4a1a7db30da P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Committing config change with OpId 1.140: config changed from index 139 to 140, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:22.691372  4305 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c1ff96ff85b4d5e9bd80fe5838b16db with delete_type TABLET_DATA_TOMBSTONED (TS 8b812251ceb44fbe8a54fea04a7f8dfc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54992
I20240503 02:12:22.691545  5635 raft_consensus.cc:2932] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Committing config change with OpId 1.140: config changed from index 139 to 140, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New config: { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:22.694927  5633 raft_consensus.cc:1261] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 143. Preceding OpId from leader: term: 1 index: 144. (index mismatch)
I20240503 02:12:22.695044  3908 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 07df29aeb41b427a8dcaf4a1a7db30da with cas_config_opid_index 139: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:22.695675  6193 consensus_queue.cc:1035] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 144, Last known committed idx: 143, Time since last communication: 0.000s
I20240503 02:12:22.699353  6363 raft_consensus.cc:2932] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.109: config changed from index 107 to 109, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:22.700328  3919 catalog_manager.cc:5484] T 07df29aeb41b427a8dcaf4a1a7db30da P 9c804fe67ee34e7fae5fab23db90f298 reported cstate change: config changed from index 139 to 140, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 140 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:22.704303  6193 raft_consensus.cc:2932] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.144: config changed from index 143 to 144, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New config: { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
W20240503 02:12:22.705052  4153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
I20240503 02:12:22.708005  6413 tablet_replica.cc:330] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc: stopping tablet replica
I20240503 02:12:22.708289  6413 raft_consensus.cc:2229] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:22.708530  6413 raft_consensus.cc:2258] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:22.708251  5633 raft_consensus.cc:2932] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Committing config change with OpId 1.144: config changed from index 143 to 144, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New config: { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:22.711469  3909 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ccccc26d94514b87a91b0c00fa18e669 with cas_config_opid_index 143: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:22.711897  6413 ts_tablet_manager.cc:1875] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:22.712065  3907 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 27609e0e32a34efc915bbff6750f8234 with cas_config_opid_index 107: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:22.714130  3920 catalog_manager.cc:5484] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index 107 to 109, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 02:12:22.716702  4153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
I20240503 02:12:22.717690  5503 raft_consensus.cc:2932] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Committing config change with OpId 1.109: config changed from index 107 to 109, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } }
I20240503 02:12:22.718264  3915 catalog_manager.cc:5484] T ccccc26d94514b87a91b0c00fa18e669 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index 143 to 144, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 144 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:22.721534  4173 tablet_service.cc:1514] Processing DeleteTablet for tablet 27609e0e32a34efc915bbff6750f8234 with delete_type TABLET_DATA_TOMBSTONED (TS d7f19d54f56849f495dc547de6a47ad2 not found in new config with opid_index 109) from {username='slave'} at 127.0.0.1:55800
W20240503 02:12:22.723999  4153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
I20240503 02:12:22.727047  6413 ts_tablet_manager.cc:1888] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 02:12:22.727280  6413 log.cc:1196] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 8b812251ceb44fbe8a54fea04a7f8dfc: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/1c1ff96ff85b4d5e9bd80fe5838b16db
I20240503 02:12:22.729519  3907 catalog_manager.cc:4830] TS 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599): tablet 1c1ff96ff85b4d5e9bd80fe5838b16db (table TestTable [id=a2f222c7915242918dc88c24b67a23ef]) successfully deleted
I20240503 02:12:22.729796  4305 tablet_service.cc:1514] Processing DeleteTablet for tablet ccccc26d94514b87a91b0c00fa18e669 with delete_type TABLET_DATA_TOMBSTONED (TS 8b812251ceb44fbe8a54fea04a7f8dfc not found in new config with opid_index 144) from {username='slave'} at 127.0.0.1:54992
I20240503 02:12:22.730134  6413 tablet_replica.cc:330] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc: stopping tablet replica
I20240503 02:12:22.730331  6413 raft_consensus.cc:2229] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:22.732234  6413 pending_rounds.cc:62] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc: Trying to abort 1 pending ops.
I20240503 02:12:22.734133  6413 pending_rounds.cc:68] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc: Aborting op as it isn't in flight: id { term: 1 index: 142 } timestamp: 7023420794974179328 op_type: WRITE_OP write_request { tablet_id: "ccccc26d94514b87a91b0c00fa18e669" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\312\257\343JI\016\342\262\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""52e1b3f03fd8b67e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023420794625941504 authz_token { token_data: "\010\251\222\321\261\006\"3\n\005slave\022*\n 908c33982a334c0a957523734a0931d3\020\001\030\001 \001(\001" signature: "<redacted>""\217\305\265\252\220\224<`E\332\305*p\341\013\270\260\214\372)\226L\365L\242\033e):\315b\202\033C\324\211\257o\033,{W\353\010`\\g\273\367\355\301\217 \276\376\307D\227\223\231Cfr\370" signing_key_seq_num: 0 } } request_id { client_id: "7ca02b32107e43e889c09f3becc08f4f" seq_no: 397 first_incomplete_seq_no: 397 attempt_no: 0 }
W20240503 02:12:22.734814  4153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
I20240503 02:12:22.735726  6413 raft_consensus.cc:2258] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:22.747870  4153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
I20240503 02:12:22.751143  4305 tablet_service.cc:1514] Processing DeleteTablet for tablet 07df29aeb41b427a8dcaf4a1a7db30da with delete_type TABLET_DATA_TOMBSTONED (TS 8b812251ceb44fbe8a54fea04a7f8dfc not found in new config with opid_index 140) from {username='slave'} at 127.0.0.1:54992
I20240503 02:12:22.751535  6414 tablet_replica.cc:330] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2: stopping tablet replica
I20240503 02:12:22.751834  6414 raft_consensus.cc:2229] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:22.753026  6414 raft_consensus.cc:2258] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:22.754348  6413 ts_tablet_manager.cc:1875] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:22.758834  6414 ts_tablet_manager.cc:1875] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:12:22.761226  4153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
I20240503 02:12:22.765316  6413 ts_tablet_manager.cc:1888] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.142
I20240503 02:12:22.765632  6413 log.cc:1196] T ccccc26d94514b87a91b0c00fa18e669 P 8b812251ceb44fbe8a54fea04a7f8dfc: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/ccccc26d94514b87a91b0c00fa18e669
I20240503 02:12:22.772872  6414 ts_tablet_manager.cc:1888] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.108
I20240503 02:12:22.776739  6414 log.cc:1196] T 27609e0e32a34efc915bbff6750f8234 P d7f19d54f56849f495dc547de6a47ad2: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/27609e0e32a34efc915bbff6750f8234
I20240503 02:12:22.778893  3908 catalog_manager.cc:4830] TS d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433): tablet 27609e0e32a34efc915bbff6750f8234 (table rebalance_test_table_4 [id=7bd39e7281604b6ebd6334a68999befc]) successfully deleted
W20240503 02:12:22.780853  4153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
W20240503 02:12:22.798377  4153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
W20240503 02:12:22.814922  4152 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
W20240503 02:12:22.833503  4152 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
W20240503 02:12:22.850437  4152 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
W20240503 02:12:22.867059  4152 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
I20240503 02:12:22.874646  3907 catalog_manager.cc:4830] TS 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599): tablet ccccc26d94514b87a91b0c00fa18e669 (table rebalance_test_table_1 [id=908c33982a334c0a957523734a0931d3]) successfully deleted
I20240503 02:12:22.876279  6413 tablet_replica.cc:330] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc: stopping tablet replica
I20240503 02:12:22.876581  6413 raft_consensus.cc:2229] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:22.878187  6413 raft_consensus.cc:2258] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:22.885530  6413 ts_tablet_manager.cc:1875] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:12:22.888175  4152 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
W20240503 02:12:22.915874  4152 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
I20240503 02:12:22.928287  6413 ts_tablet_manager.cc:1888] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.139
I20240503 02:12:22.931409  6413 log.cc:1196] T 07df29aeb41b427a8dcaf4a1a7db30da P 8b812251ceb44fbe8a54fea04a7f8dfc: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/07df29aeb41b427a8dcaf4a1a7db30da
I20240503 02:12:22.932632  3907 catalog_manager.cc:4830] TS 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599): tablet 07df29aeb41b427a8dcaf4a1a7db30da (table rebalance_test_table_3 [id=76e610e6ec844c2da65036194e355bbe]) successfully deleted
W20240503 02:12:22.943475  4152 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Service unavailable: leader transfer in progress
I20240503 02:12:22.949661  6031 raft_consensus.cc:979] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2: : Instructing follower c4af56cee8e546c384cf20f40deeacd7 to start an election
I20240503 02:12:22.949878  6031 raft_consensus.cc:1067] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Signalling peer c4af56cee8e546c384cf20f40deeacd7 to start an election
I20240503 02:12:22.951090  4061 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "75a2da7ca68b43e28cad0847e4d27982"
dest_uuid: "c4af56cee8e546c384cf20f40deeacd7"
 from {username='slave'} at 127.0.145.2:39407
I20240503 02:12:22.951393  4061 raft_consensus.cc:482] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:22.951541  4061 raft_consensus.cc:3037] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:22.954499  4061 raft_consensus.cc:504] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:22.955664  4061 leader_election.cc:288] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 2 election: Requested vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433), 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7:35457)
I20240503 02:12:22.957170  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a2da7ca68b43e28cad0847e4d27982" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 2 candidate_status { last_received { term: 1 index: 161 } } ignore_live_leader: true dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
I20240503 02:12:22.957427  4194 raft_consensus.cc:3032] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:22.957509  4194 raft_consensus.cc:729] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Running, Role: LEADER
I20240503 02:12:22.957823  4194 consensus_queue.cc:260] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 161, Committed index: 161, Last appended: 1.161, Last appended by leader: 161, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:22.958213  4194 raft_consensus.cc:3037] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:22.961846  4194 raft_consensus.cc:2454] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 2.
I20240503 02:12:22.962589  3998 leader_election.cc:302] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4af56cee8e546c384cf20f40deeacd7, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:22.962949  6194 raft_consensus.cc:2781] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:22.963109  6194 raft_consensus.cc:686] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 2 LEADER]: Becoming Leader. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Running, Role: LEADER
I20240503 02:12:22.963392  6194 consensus_queue.cc:237] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 161, Committed index: 161, Last appended: 1.161, Last appended by leader: 161, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:22.966178  3919 catalog_manager.cc:5484] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: term changed from 1 to 2, leader changed from d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) to c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1). New cstate: current_term: 2 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:22.967159  4152 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53722: Illegal state: replica d7f19d54f56849f495dc547de6a47ad2 is not leader of this config: current role FOLLOWER
I20240503 02:12:22.986465  5763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a2da7ca68b43e28cad0847e4d27982" candidate_uuid: "c4af56cee8e546c384cf20f40deeacd7" candidate_term: 2 candidate_status { last_received { term: 1 index: 161 } } ignore_live_leader: true dest_uuid: "6c3559ccff5d4c6681b3c90f892143dc"
I20240503 02:12:22.987334  5763 raft_consensus.cc:3037] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:23.065555  5763 raft_consensus.cc:2454] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4af56cee8e546c384cf20f40deeacd7 in term 2.
I20240503 02:12:23.146428  4061 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e9898815c904092af0a6f238ed961bd"
dest_uuid: "c4af56cee8e546c384cf20f40deeacd7"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:56000
I20240503 02:12:23.146709  4061 raft_consensus.cc:595] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:23.147836  6194 raft_consensus.cc:979] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7: : Instructing follower d7f19d54f56849f495dc547de6a47ad2 to start an election
I20240503 02:12:23.148033  6194 raft_consensus.cc:1067] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Signalling peer d7f19d54f56849f495dc547de6a47ad2 to start an election
W20240503 02:12:23.150178  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42610: Service unavailable: leader transfer in progress
I20240503 02:12:23.150233  4194 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e9898815c904092af0a6f238ed961bd"
dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
 from {username='slave'} at 127.0.145.1:38881
I20240503 02:12:23.150490  4194 raft_consensus.cc:482] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:23.150657  4194 raft_consensus.cc:3037] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:23.153242  4194 raft_consensus.cc:504] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } }
I20240503 02:12:23.155143  4193 raft_consensus.cc:1226] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Rejecting Update request from peer c4af56cee8e546c384cf20f40deeacd7 for earlier term 1. Current term is 2. Ops: []
I20240503 02:12:23.157303  6193 consensus_queue.cc:1046] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 }, Status: INVALID_TERM, Last received: 1.131, Next index: 132, Last known committed idx: 130, Time since last communication: 0.000s
I20240503 02:12:23.157753  6193 raft_consensus.cc:3032] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:23.157778  4061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9898815c904092af0a6f238ed961bd" candidate_uuid: "d7f19d54f56849f495dc547de6a47ad2" candidate_term: 2 candidate_status { last_received { term: 1 index: 131 } } ignore_live_leader: true dest_uuid: "c4af56cee8e546c384cf20f40deeacd7"
I20240503 02:12:23.157905  6193 raft_consensus.cc:729] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Running, Role: LEADER
I20240503 02:12:23.158300  6193 consensus_queue.cc:260] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 131, Committed index: 131, Last appended: 1.131, Last appended by leader: 131, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } }
I20240503 02:12:23.158795  6193 raft_consensus.cc:3037] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:23.162673  4194 leader_election.cc:288] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 2 election: Requested vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835), 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4:33659)
I20240503 02:12:23.163677  5374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9898815c904092af0a6f238ed961bd" candidate_uuid: "d7f19d54f56849f495dc547de6a47ad2" candidate_term: 2 candidate_status { last_received { term: 1 index: 131 } } ignore_live_leader: true dest_uuid: "6ebeaff495e042c78067c5b18600dc98"
I20240503 02:12:23.164027  5374 raft_consensus.cc:3037] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:23.166411  5374 raft_consensus.cc:2454] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f19d54f56849f495dc547de6a47ad2 in term 2.
I20240503 02:12:23.166910  4130 leader_election.cc:302] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ebeaff495e042c78067c5b18600dc98, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:23.167275  6394 raft_consensus.cc:2781] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:23.167415  6394 raft_consensus.cc:686] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 2 LEADER]: Becoming Leader. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Running, Role: LEADER
I20240503 02:12:23.167740  6394 consensus_queue.cc:237] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 1.131, Last appended by leader: 131, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } }
I20240503 02:12:23.171564  3919 catalog_manager.cc:5484] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: term changed from 1 to 2, leader changed from c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) to d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2). New cstate: current_term: 2 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:23.174067  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42610: Illegal state: replica c4af56cee8e546c384cf20f40deeacd7 is not leader of this config: current role FOLLOWER
I20240503 02:12:23.183415  4061 raft_consensus.cc:2454] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f19d54f56849f495dc547de6a47ad2 in term 2.
I20240503 02:12:23.183439  5764 raft_consensus.cc:1261] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [term 2 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 161. Preceding OpId from leader: term: 2 index: 163. (index mismatch)
I20240503 02:12:23.187238  4194 raft_consensus.cc:1261] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 161. Preceding OpId from leader: term: 2 index: 163. (index mismatch)
I20240503 02:12:23.188573  6396 consensus_queue.cc:1035] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 162, Last known committed idx: 161, Time since last communication: 0.004s
I20240503 02:12:23.189083  6193 consensus_queue.cc:1035] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 162, Last known committed idx: 161, Time since last communication: 0.000s
I20240503 02:12:23.217967  5374 raft_consensus.cc:1261] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 2 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 131. Preceding OpId from leader: term: 2 index: 133. (index mismatch)
I20240503 02:12:23.218756  6166 consensus_queue.cc:1035] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I20240503 02:12:23.221717  4061 raft_consensus.cc:1261] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 131. Preceding OpId from leader: term: 2 index: 133. (index mismatch)
I20240503 02:12:23.225692  6394 consensus_queue.cc:1035] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I20240503 02:12:23.233233  6163 mvcc.cc:205] Tried to move back new op lower bound from 7023420797801295872 to 7023420797617901568. Current Snapshot: MvccSnapshot[applied={T|T < 7023420797801295872 or (T in {7023420797801295872})}]
I20240503 02:12:23.255462  4061 consensus_queue.cc:237] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 165, Committed index: 165, Last appended: 2.165, Last appended by leader: 161, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:23.258834  5764 raft_consensus.cc:1261] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [term 2 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 2 index: 163. Preceding OpId from leader: term: 2 index: 166. (index mismatch)
I20240503 02:12:23.259485  6396 consensus_queue.cc:1035] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 161, Time since last communication: 0.000s
I20240503 02:12:23.262454  6332 mvcc.cc:205] Tried to move back new op lower bound from 7023420797660073984 to 7023420796780175360. Current Snapshot: MvccSnapshot[applied={T|T < 7023420797660073984}]
I20240503 02:12:23.280010  5763 raft_consensus.cc:1203] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.163->[2.164-2.165]   Dedup: 2.165->[]
I20240503 02:12:23.286345  6396 raft_consensus.cc:2932] T 75a2da7ca68b43e28cad0847e4d27982 P c4af56cee8e546c384cf20f40deeacd7 [term 2 LEADER]: Committing config change with OpId 2.166: config changed from index 134 to 166, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:23.287528  5763 raft_consensus.cc:2932] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [term 2 FOLLOWER]: Committing config change with OpId 2.166: config changed from index 134 to 166, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:23.296190  3915 catalog_manager.cc:5484] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc reported cstate change: config changed from index 134 to 166, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New cstate: current_term: 2 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:23.298954  3909 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 75a2da7ca68b43e28cad0847e4d27982 with cas_config_opid_index 134: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:23.316098  4193 consensus_queue.cc:237] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 134, Committed index: 134, Last appended: 2.134, Last appended by leader: 131, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } }
I20240503 02:12:23.317030  4173 tablet_service.cc:1514] Processing DeleteTablet for tablet 75a2da7ca68b43e28cad0847e4d27982 with delete_type TABLET_DATA_TOMBSTONED (TS d7f19d54f56849f495dc547de6a47ad2 not found in new config with opid_index 166) from {username='slave'} at 127.0.0.1:55800
I20240503 02:12:23.321020  5372 raft_consensus.cc:1261] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 2 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 2 index: 134. Preceding OpId from leader: term: 2 index: 135. (index mismatch)
I20240503 02:12:23.324404  6031 consensus_queue.cc:1035] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 134, Time since last communication: 0.000s
I20240503 02:12:23.333086  6394 raft_consensus.cc:2932] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 2 LEADER]: Committing config change with OpId 2.135: config changed from index 88 to 135, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:23.336567  5374 raft_consensus.cc:2932] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 2 FOLLOWER]: Committing config change with OpId 2.135: config changed from index 88 to 135, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:23.369563  3908 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e9898815c904092af0a6f238ed961bd with cas_config_opid_index 88: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:23.369175  3915 catalog_manager.cc:5484] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: config changed from index 88 to 135, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New cstate: current_term: 2 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:23.375382  4061 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "368fca99d20d4d6e826c45e44379c056"
dest_uuid: "c4af56cee8e546c384cf20f40deeacd7"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:56016
I20240503 02:12:23.375685  4061 raft_consensus.cc:595] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:23.378899  4043 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e9898815c904092af0a6f238ed961bd with delete_type TABLET_DATA_TOMBSTONED (TS c4af56cee8e546c384cf20f40deeacd7 not found in new config with opid_index 135) from {username='slave'} at 127.0.0.1:51996
W20240503 02:12:23.395951  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
W20240503 02:12:23.402209  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
I20240503 02:12:23.403538  6450 tablet_replica.cc:330] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2: stopping tablet replica
I20240503 02:12:23.407676  6452 tablet_replica.cc:330] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7: stopping tablet replica
I20240503 02:12:23.411190  6452 raft_consensus.cc:2229] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.413307  6452 raft_consensus.cc:2258] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:23.417135  6452 ts_tablet_manager.cc:1875] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:23.419598  6450 raft_consensus.cc:2229] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:23.420089  6450 raft_consensus.cc:2258] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240503 02:12:23.420866  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
I20240503 02:12:23.422452  6450 ts_tablet_manager.cc:1875] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:12:23.431944  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
W20240503 02:12:23.442807  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
W20240503 02:12:23.452666  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
I20240503 02:12:23.454931  6452 ts_tablet_manager.cc:1888] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.134
I20240503 02:12:23.455076  6452 log.cc:1196] T 6e9898815c904092af0a6f238ed961bd P c4af56cee8e546c384cf20f40deeacd7: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/6e9898815c904092af0a6f238ed961bd
I20240503 02:12:23.458536  3909 catalog_manager.cc:4830] TS c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835): tablet 6e9898815c904092af0a6f238ed961bd (table rebalance_test_table_4 [id=7bd39e7281604b6ebd6334a68999befc]) successfully deleted
W20240503 02:12:23.477499  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
W20240503 02:12:23.488201  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
I20240503 02:12:23.489373  6450 ts_tablet_manager.cc:1888] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.165
I20240503 02:12:23.489531  6450 log.cc:1196] T 75a2da7ca68b43e28cad0847e4d27982 P d7f19d54f56849f495dc547de6a47ad2: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/75a2da7ca68b43e28cad0847e4d27982
I20240503 02:12:23.490999  3908 catalog_manager.cc:4830] TS d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433): tablet 75a2da7ca68b43e28cad0847e4d27982 (table rebalance_test_table_0 [id=3a36c423c3f74ce392ff73837341d077]) successfully deleted
W20240503 02:12:23.500829  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
W20240503 02:12:23.514178  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
W20240503 02:12:23.526724  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
W20240503 02:12:23.541682  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
W20240503 02:12:23.559901  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
W20240503 02:12:23.579242  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
W20240503 02:12:23.597235  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
W20240503 02:12:23.615317  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
W20240503 02:12:23.634687  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
I20240503 02:12:23.648903  6194 raft_consensus.cc:979] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7: : Instructing follower d7f19d54f56849f495dc547de6a47ad2 to start an election
I20240503 02:12:23.649111  6194 raft_consensus.cc:1067] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Signalling peer d7f19d54f56849f495dc547de6a47ad2 to start an election
I20240503 02:12:23.650051  4193 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "368fca99d20d4d6e826c45e44379c056"
dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
 from {username='slave'} at 127.0.145.1:38881
I20240503 02:12:23.650311  4193 raft_consensus.cc:482] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:23.650444  4193 raft_consensus.cc:3037] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:23.653211  4193 raft_consensus.cc:504] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:23.653963  4193 leader_election.cc:288] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 2 election: Requested vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835), 6c3559ccff5d4c6681b3c90f892143dc (127.0.145.7:35457)
I20240503 02:12:23.655007  5763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368fca99d20d4d6e826c45e44379c056" candidate_uuid: "d7f19d54f56849f495dc547de6a47ad2" candidate_term: 2 candidate_status { last_received { term: 1 index: 96 } } ignore_live_leader: true dest_uuid: "6c3559ccff5d4c6681b3c90f892143dc"
I20240503 02:12:23.655196  4061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368fca99d20d4d6e826c45e44379c056" candidate_uuid: "d7f19d54f56849f495dc547de6a47ad2" candidate_term: 2 candidate_status { last_received { term: 1 index: 96 } } ignore_live_leader: true dest_uuid: "c4af56cee8e546c384cf20f40deeacd7"
I20240503 02:12:23.655270  5763 raft_consensus.cc:3037] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:23.655405  4061 raft_consensus.cc:3032] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:23.655512  4061 raft_consensus.cc:729] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Running, Role: LEADER
W20240503 02:12:23.655544  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Service unavailable: leader transfer in progress
I20240503 02:12:23.655831  4061 consensus_queue.cc:260] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 1.96, Last appended by leader: 96, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:23.656231  4061 raft_consensus.cc:3037] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:23.658341  5763 raft_consensus.cc:2454] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f19d54f56849f495dc547de6a47ad2 in term 2.
I20240503 02:12:23.665673  4061 raft_consensus.cc:2454] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f19d54f56849f495dc547de6a47ad2 in term 2.
I20240503 02:12:23.659042  4130 leader_election.cc:302] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c3559ccff5d4c6681b3c90f892143dc, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:23.667500  6031 raft_consensus.cc:2781] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:23.667690  6031 raft_consensus.cc:686] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 2 LEADER]: Becoming Leader. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Running, Role: LEADER
I20240503 02:12:23.667985  6031 consensus_queue.cc:237] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 1.96, Last appended by leader: 96, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:23.673691  3919 catalog_manager.cc:5484] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: term changed from 1 to 2, leader changed from c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) to d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2). New cstate: current_term: 2 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:23.676642  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42584: Illegal state: replica c4af56cee8e546c384cf20f40deeacd7 is not leader of this config: current role FOLLOWER
I20240503 02:12:23.784039  4061 consensus_queue.cc:237] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 144, Committed index: 144, Last appended: 1.144, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } }
W20240503 02:12:23.786172  3999 consensus_peers.cc:485] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 -> Peer 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4:33659): Couldn't send request to peer 6ebeaff495e042c78067c5b18600dc98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87e472f1975b4141b795aa6c3b75c7c0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:23.787436  4324 raft_consensus.cc:1261] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 144. Preceding OpId from leader: term: 1 index: 145. (index mismatch)
I20240503 02:12:23.788312  6396 consensus_queue.cc:1035] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 145, Last known committed idx: 144, Time since last communication: 0.000s
I20240503 02:12:23.791726  6396 raft_consensus.cc:2932] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.145: config changed from index -1 to 145, NON_VOTER 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) added. New config: { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } } }
I20240503 02:12:23.792200  4324 raft_consensus.cc:2932] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Committing config change with OpId 1.145: config changed from index -1 to 145, NON_VOTER 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) added. New config: { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } } }
I20240503 02:12:23.801959  5764 raft_consensus.cc:1261] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc [term 2 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 2 index: 98. (index mismatch)
I20240503 02:12:23.801981  4061 raft_consensus.cc:1261] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 2 index: 98. (index mismatch)
I20240503 02:12:23.803442  6031 consensus_queue.cc:1035] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20240503 02:12:23.803974  6031 consensus_queue.cc:1035] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20240503 02:12:23.810098  3919 catalog_manager.cc:5484] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index -1 to 145, NON_VOTER 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) added. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } } }
I20240503 02:12:23.941583  4193 consensus_queue.cc:237] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 2.101, Last appended by leader: 96, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:23.948186  5765 raft_consensus.cc:1261] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc [term 2 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 2 index: 101. Preceding OpId from leader: term: 2 index: 102. (index mismatch)
I20240503 02:12:23.951359  6394 consensus_queue.cc:1035] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20240503 02:12:23.971274  6394 raft_consensus.cc:2932] T 368fca99d20d4d6e826c45e44379c056 P d7f19d54f56849f495dc547de6a47ad2 [term 2 LEADER]: Committing config change with OpId 2.102: config changed from index 77 to 102, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:23.979544  5763 raft_consensus.cc:2932] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc [term 2 FOLLOWER]: Committing config change with OpId 2.102: config changed from index 77 to 102, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:23.986403  3919 catalog_manager.cc:5484] T 368fca99d20d4d6e826c45e44379c056 P 6c3559ccff5d4c6681b3c90f892143dc reported cstate change: config changed from index 77 to 102, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New cstate: current_term: 2 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } } }
I20240503 02:12:23.990918  4043 tablet_service.cc:1514] Processing DeleteTablet for tablet 368fca99d20d4d6e826c45e44379c056 with delete_type TABLET_DATA_TOMBSTONED (TS c4af56cee8e546c384cf20f40deeacd7 not found in new config with opid_index 102) from {username='slave'} at 127.0.0.1:51996
I20240503 02:12:23.991537  3908 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 368fca99d20d4d6e826c45e44379c056 with cas_config_opid_index 77: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:24.004545  6495 tablet_replica.cc:330] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7: stopping tablet replica
I20240503 02:12:24.004930  6495 raft_consensus.cc:2229] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:24.005334  6495 raft_consensus.cc:2258] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:24.006381  6495 ts_tablet_manager.cc:1875] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:24.028288  6495 ts_tablet_manager.cc:1888] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.101
I20240503 02:12:24.028545  6495 log.cc:1196] T 368fca99d20d4d6e826c45e44379c056 P c4af56cee8e546c384cf20f40deeacd7: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/368fca99d20d4d6e826c45e44379c056
I20240503 02:12:24.032140  3909 catalog_manager.cc:4830] TS c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835): tablet 368fca99d20d4d6e826c45e44379c056 (table rebalance_test_table_2 [id=46a3ef9b51d44a84a25ae9ff7198c9b5]) successfully deleted
I20240503 02:12:24.252735  6513 ts_tablet_manager.cc:902] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98: Initiating tablet copy from peer c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:24.274032  6513 tablet_copy_client.cc:308] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:43835
I20240503 02:12:24.275391  4083 tablet_copy_service.cc:140] P c4af56cee8e546c384cf20f40deeacd7: Received BeginTabletCopySession request for tablet 87e472f1975b4141b795aa6c3b75c7c0 from peer 6ebeaff495e042c78067c5b18600dc98 ({username='slave'} at 127.0.145.4:41257)
I20240503 02:12:24.275592  4083 tablet_copy_service.cc:161] P c4af56cee8e546c384cf20f40deeacd7: Beginning new tablet copy session on tablet 87e472f1975b4141b795aa6c3b75c7c0 from peer 6ebeaff495e042c78067c5b18600dc98 at {username='slave'} at 127.0.145.4:41257: session id = 6ebeaff495e042c78067c5b18600dc98-87e472f1975b4141b795aa6c3b75c7c0
I20240503 02:12:24.278553  4083 tablet_copy_source_session.cc:205] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:24.288169  6513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e472f1975b4141b795aa6c3b75c7c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:24.312986  6513 tablet_copy_client.cc:783] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:24.313499  6513 tablet_copy_client.cc:647] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:24.378063  6513 tablet_copy_client.cc:523] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:24.393536  6513 tablet_bootstrap.cc:492] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98: Bootstrap starting.
W20240503 02:12:24.477622  4261 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.7:35457, user_credentials={real_user=slave}} blocked reactor thread for 48219.7us
I20240503 02:12:25.079850  6513 tablet_bootstrap.cc:492] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98: Bootstrap replayed 1/1 log segments. Stats: ops{read=156 overwritten=0 applied=156 ignored=0} inserts{seen=841 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:25.080492  6513 tablet_bootstrap.cc:492] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98: Bootstrap complete.
I20240503 02:12:25.089560  6513 ts_tablet_manager.cc:1367] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98: Time spent bootstrapping tablet: real 0.691s	user 0.242s	sys 0.018s
I20240503 02:12:25.099052  6513 raft_consensus.cc:348] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } }
I20240503 02:12:25.100697  6513 raft_consensus.cc:729] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ebeaff495e042c78067c5b18600dc98, State: Initialized, Role: LEARNER
I20240503 02:12:25.105218  6513 consensus_queue.cc:260] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 156, Last appended: 1.156, Last appended by leader: 156, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: true } }
I20240503 02:12:25.120987  6513 ts_tablet_manager.cc:1398] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20240503 02:12:25.131258  4083 tablet_copy_service.cc:342] P c4af56cee8e546c384cf20f40deeacd7: Request end of tablet copy session 6ebeaff495e042c78067c5b18600dc98-87e472f1975b4141b795aa6c3b75c7c0 received from {username='slave'} at 127.0.145.4:41257
I20240503 02:12:25.131457  4083 tablet_copy_service.cc:433] P c4af56cee8e546c384cf20f40deeacd7: ending tablet copy session 6ebeaff495e042c78067c5b18600dc98-87e472f1975b4141b795aa6c3b75c7c0 on tablet 87e472f1975b4141b795aa6c3b75c7c0 with peer 6ebeaff495e042c78067c5b18600dc98
I20240503 02:12:25.716835  5374 raft_consensus.cc:1203] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.144->[1.145-1.186]   Dedup: 1.156->[1.157-1.186]
I20240503 02:12:25.758847  6194 raft_consensus.cc:1050] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7: attempting to promote NON_VOTER 6ebeaff495e042c78067c5b18600dc98 to VOTER
I20240503 02:12:25.759644  6194 consensus_queue.cc:237] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 1.187, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } }
I20240503 02:12:25.763552  4324 raft_consensus.cc:1261] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 187. Preceding OpId from leader: term: 1 index: 188. (index mismatch)
I20240503 02:12:25.764169  6396 consensus_queue.cc:1035] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 188, Last known committed idx: 187, Time since last communication: 0.000s
I20240503 02:12:25.765677  5372 raft_consensus.cc:1261] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98 [term 1 LEARNER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 187. Preceding OpId from leader: term: 1 index: 188. (index mismatch)
I20240503 02:12:25.767462  6396 raft_consensus.cc:2932] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.188: config changed from index 145 to 188, 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:25.767767  4324 raft_consensus.cc:2932] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Committing config change with OpId 1.188: config changed from index 145 to 188, 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:25.770522  5924 consensus_queue.cc:1035] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 188, Last known committed idx: 187, Time since last communication: 0.000s
I20240503 02:12:25.776358  3915 catalog_manager.cc:5484] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index 145 to 188, 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:25.816926  5374 raft_consensus.cc:2932] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98 [term 1 FOLLOWER]: Committing config change with OpId 1.188: config changed from index 145 to 188, 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:26.189055  4062 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "87e472f1975b4141b795aa6c3b75c7c0"
dest_uuid: "c4af56cee8e546c384cf20f40deeacd7"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:56112
I20240503 02:12:26.189308  4062 raft_consensus.cc:595] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Received request to transfer leadership
W20240503 02:12:26.250667  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.254104  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.259447  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.270102  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.283035  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.293371  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.304533  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.315867  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.329094  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.345654  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.361478  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.377444  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.394373  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.415530  4021 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.437923  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
W20240503 02:12:26.460150  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
I20240503 02:12:26.475641  6194 raft_consensus.cc:979] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7: : Instructing follower 8b812251ceb44fbe8a54fea04a7f8dfc to start an election
I20240503 02:12:26.475852  6194 raft_consensus.cc:1067] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Signalling peer 8b812251ceb44fbe8a54fea04a7f8dfc to start an election
I20240503 02:12:26.477542  4322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87e472f1975b4141b795aa6c3b75c7c0"
dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc"
 from {username='slave'} at 127.0.145.1:47175
I20240503 02:12:26.477823  4322 raft_consensus.cc:482] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:26.477929  4322 raft_consensus.cc:3037] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:26.480921  4322 raft_consensus.cc:504] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } }
W20240503 02:12:26.482925  4023 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Service unavailable: leader transfer in progress
I20240503 02:12:26.486843  4322 leader_election.cc:288] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 2 election: Requested vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835), 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4:33659)
I20240503 02:12:26.486891  4062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e472f1975b4141b795aa6c3b75c7c0" candidate_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" candidate_term: 2 candidate_status { last_received { term: 1 index: 196 } } ignore_live_leader: true dest_uuid: "c4af56cee8e546c384cf20f40deeacd7"
I20240503 02:12:26.487092  4062 raft_consensus.cc:3032] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:26.487190  4062 raft_consensus.cc:729] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Running, Role: LEADER
I20240503 02:12:26.487475  4062 consensus_queue.cc:260] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 196, Committed index: 196, Last appended: 1.196, Last appended by leader: 196, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } }
I20240503 02:12:26.487540  5373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e472f1975b4141b795aa6c3b75c7c0" candidate_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" candidate_term: 2 candidate_status { last_received { term: 1 index: 196 } } ignore_live_leader: true dest_uuid: "6ebeaff495e042c78067c5b18600dc98"
I20240503 02:12:26.487773  4062 raft_consensus.cc:3037] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:26.487802  5373 raft_consensus.cc:3037] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:26.490178  4062 raft_consensus.cc:2454] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b812251ceb44fbe8a54fea04a7f8dfc in term 2.
I20240503 02:12:26.490443  5373 raft_consensus.cc:2454] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b812251ceb44fbe8a54fea04a7f8dfc in term 2.
I20240503 02:12:26.490698  4261 leader_election.cc:302] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b812251ceb44fbe8a54fea04a7f8dfc, c4af56cee8e546c384cf20f40deeacd7; no voters: 
I20240503 02:12:26.491114  6545 raft_consensus.cc:2781] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:26.491261  6545 raft_consensus.cc:686] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 2 LEADER]: Becoming Leader. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Running, Role: LEADER
I20240503 02:12:26.491611  6545 consensus_queue.cc:237] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 196, Committed index: 196, Last appended: 1.196, Last appended by leader: 196, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } }
I20240503 02:12:26.495291  3919 catalog_manager.cc:5484] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: term changed from 1 to 2, leader changed from c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) to 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3). New cstate: current_term: 2 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 188 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:26.505789  4022 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42600: Illegal state: replica c4af56cee8e546c384cf20f40deeacd7 is not leader of this config: current role FOLLOWER
I20240503 02:12:26.580348  5372 raft_consensus.cc:1261] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98 [term 2 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 196. Preceding OpId from leader: term: 2 index: 198. (index mismatch)
I20240503 02:12:26.580577  4062 raft_consensus.cc:1261] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 196. Preceding OpId from leader: term: 2 index: 198. (index mismatch)
I20240503 02:12:26.581238  6363 consensus_queue.cc:1035] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 197, Last known committed idx: 196, Time since last communication: 0.000s
I20240503 02:12:26.588016  6363 consensus_queue.cc:1035] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 197, Last known committed idx: 196, Time since last communication: 0.000s
I20240503 02:12:26.594386  6335 mvcc.cc:205] Tried to move back new op lower bound from 7023420811588001792 to 7023420811236380672. Current Snapshot: MvccSnapshot[applied={T|T < 7023420811588001792}]
I20240503 02:12:26.635425  4322 consensus_queue.cc:237] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 200, Committed index: 200, Last appended: 2.200, Last appended by leader: 196, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } }
I20240503 02:12:26.648418  5373 raft_consensus.cc:1261] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98 [term 2 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 2 index: 200. Preceding OpId from leader: term: 2 index: 201. (index mismatch)
I20240503 02:12:26.649014  6363 consensus_queue.cc:1035] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 201, Last known committed idx: 200, Time since last communication: 0.000s
I20240503 02:12:26.655758  6545 raft_consensus.cc:2932] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 2 LEADER]: Committing config change with OpId 2.201: config changed from index 188 to 201, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:26.657148  5373 raft_consensus.cc:2932] T 87e472f1975b4141b795aa6c3b75c7c0 P 6ebeaff495e042c78067c5b18600dc98 [term 2 FOLLOWER]: Committing config change with OpId 2.201: config changed from index 188 to 201, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } }
I20240503 02:12:26.661446  3907 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 87e472f1975b4141b795aa6c3b75c7c0 with cas_config_opid_index 188: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:26.663178  3919 catalog_manager.cc:5484] T 87e472f1975b4141b795aa6c3b75c7c0 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index 188 to 201, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New cstate: current_term: 2 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:26.667991  4043 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e472f1975b4141b795aa6c3b75c7c0 with delete_type TABLET_DATA_TOMBSTONED (TS c4af56cee8e546c384cf20f40deeacd7 not found in new config with opid_index 201) from {username='slave'} at 127.0.0.1:51996
I20240503 02:12:26.681195  6606 tablet_replica.cc:330] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7: stopping tablet replica
I20240503 02:12:26.681710  6606 raft_consensus.cc:2229] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:26.682137  6606 raft_consensus.cc:2258] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:26.685276  6606 ts_tablet_manager.cc:1875] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:26.699594  6606 ts_tablet_manager.cc:1888] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.200
I20240503 02:12:26.699868  6606 log.cc:1196] T 87e472f1975b4141b795aa6c3b75c7c0 P c4af56cee8e546c384cf20f40deeacd7: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/87e472f1975b4141b795aa6c3b75c7c0
I20240503 02:12:26.708915  3909 catalog_manager.cc:4830] TS c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835): tablet 87e472f1975b4141b795aa6c3b75c7c0 (table rebalance_test_table_3 [id=76e610e6ec844c2da65036194e355bbe]) successfully deleted
I20240503 02:12:27.515736  5745 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:27.540241  5485 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:27.575695  5354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:27.635362  4305 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:27.657626  4173 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:27.684214  4043 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:27.707091  5615 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:28.142225  4193 consensus_queue.cc:237] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 2.214, Last appended by leader: 131, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
I20240503 02:12:28.145967  5372 raft_consensus.cc:1261] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 2 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 2 index: 213. Preceding OpId from leader: term: 2 index: 215. (index mismatch)
I20240503 02:12:28.149241  6451 consensus_queue.cc:1035] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 213, Time since last communication: 0.000s
W20240503 02:12:28.154042  4129 consensus_peers.cc:485] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 -> Peer 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6:36423): Couldn't send request to peer 9c804fe67ee34e7fae5fab23db90f298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e9898815c904092af0a6f238ed961bd. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:28.156121  6451 raft_consensus.cc:2932] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 2 LEADER]: Committing config change with OpId 2.215: config changed from index 135 to 215, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } } }
I20240503 02:12:28.159672  5371 raft_consensus.cc:2932] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 2 FOLLOWER]: Committing config change with OpId 2.215: config changed from index 135 to 215, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } } }
I20240503 02:12:28.168768  3920 catalog_manager.cc:5484] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: config changed from index 135 to 215, NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) added. New cstate: current_term: 2 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:28.617494  6694 ts_tablet_manager.cc:902] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298: Initiating tablet copy from peer d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433)
I20240503 02:12:28.675174  6694 tablet_copy_client.cc:308] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:44433
I20240503 02:12:28.676435  4214 tablet_copy_service.cc:140] P d7f19d54f56849f495dc547de6a47ad2: Received BeginTabletCopySession request for tablet 6e9898815c904092af0a6f238ed961bd from peer 9c804fe67ee34e7fae5fab23db90f298 ({username='slave'} at 127.0.145.6:50551)
I20240503 02:12:28.676626  4214 tablet_copy_service.cc:161] P d7f19d54f56849f495dc547de6a47ad2: Beginning new tablet copy session on tablet 6e9898815c904092af0a6f238ed961bd from peer 9c804fe67ee34e7fae5fab23db90f298 at {username='slave'} at 127.0.145.6:50551: session id = 9c804fe67ee34e7fae5fab23db90f298-6e9898815c904092af0a6f238ed961bd
I20240503 02:12:28.690374  4214 tablet_copy_source_session.cc:205] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:28.692132  6694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e9898815c904092af0a6f238ed961bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.704372  6694 tablet_copy_client.cc:783] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:28.705359  6694 tablet_copy_client.cc:647] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:28.708482  6694 tablet_copy_client.cc:523] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:28.712327  6694 tablet_bootstrap.cc:492] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap starting.
I20240503 02:12:29.117146  6694 tablet_bootstrap.cc:492] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap replayed 1/1 log segments. Stats: ops{read=225 overwritten=0 applied=225 ignored=0} inserts{seen=906 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:29.117787  6694 tablet_bootstrap.cc:492] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298: Bootstrap complete.
I20240503 02:12:29.118237  6694 ts_tablet_manager.cc:1367] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298: Time spent bootstrapping tablet: real 0.406s	user 0.289s	sys 0.029s
I20240503 02:12:29.119315  6694 raft_consensus.cc:348] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
I20240503 02:12:29.120285  6694 raft_consensus.cc:729] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9c804fe67ee34e7fae5fab23db90f298, State: Initialized, Role: LEARNER
I20240503 02:12:29.120612  6694 consensus_queue.cc:260] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 225, Last appended: 2.225, Last appended by leader: 225, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: true } }
I20240503 02:12:29.136030  6694 ts_tablet_manager.cc:1398] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298: Time spent starting tablet: real 0.018s	user 0.000s	sys 0.002s
I20240503 02:12:29.137497  4214 tablet_copy_service.cc:342] P d7f19d54f56849f495dc547de6a47ad2: Request end of tablet copy session 9c804fe67ee34e7fae5fab23db90f298-6e9898815c904092af0a6f238ed961bd received from {username='slave'} at 127.0.145.6:50551
I20240503 02:12:29.137627  4214 tablet_copy_service.cc:433] P d7f19d54f56849f495dc547de6a47ad2: ending tablet copy session 9c804fe67ee34e7fae5fab23db90f298-6e9898815c904092af0a6f238ed961bd on tablet 6e9898815c904092af0a6f238ed961bd with peer 9c804fe67ee34e7fae5fab23db90f298
I20240503 02:12:29.227103  5633 raft_consensus.cc:1203] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.214->[2.215-2.236]   Dedup: 2.225->[2.226-2.236]
I20240503 02:12:29.260044  6166 raft_consensus.cc:1050] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2: attempting to promote NON_VOTER 9c804fe67ee34e7fae5fab23db90f298 to VOTER
I20240503 02:12:29.260756  6166 consensus_queue.cc:237] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 2.237, Last appended by leader: 131, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } }
I20240503 02:12:29.263856  5635 raft_consensus.cc:1261] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [term 2 LEARNER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 2 index: 237. Preceding OpId from leader: term: 2 index: 238. (index mismatch)
I20240503 02:12:29.264137  5372 raft_consensus.cc:1261] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 2 FOLLOWER]: Refusing update from remote peer d7f19d54f56849f495dc547de6a47ad2: Log matching property violated. Preceding OpId in replica: term: 2 index: 237. Preceding OpId from leader: term: 2 index: 238. (index mismatch)
I20240503 02:12:29.264708  6451 consensus_queue.cc:1035] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20240503 02:12:29.265640  6451 consensus_queue.cc:1035] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.001s
I20240503 02:12:29.268623  6672 raft_consensus.cc:2932] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 2 LEADER]: Committing config change with OpId 2.238: config changed from index 215 to 238, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:29.274300  5372 raft_consensus.cc:2932] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 2 FOLLOWER]: Committing config change with OpId 2.238: config changed from index 215 to 238, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:29.275756  3918 catalog_manager.cc:5484] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: config changed from index 215 to 238, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d7f19d54f56849f495dc547de6a47ad2" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:29.300995  5635 raft_consensus.cc:2932] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [term 2 FOLLOWER]: Committing config change with OpId 2.238: config changed from index 215 to 238, 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:29.649156  4193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e9898815c904092af0a6f238ed961bd"
dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33942
I20240503 02:12:29.649408  4193 raft_consensus.cc:595] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 2 LEADER]: Received request to transfer leadership
W20240503 02:12:29.703831  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.708670  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.718732  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.728169  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.738318  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.751796  4152 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.764166  4152 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.777447  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.789948  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.811950  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.828457  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.844038  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.865238  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.885488  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.910661  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.928937  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:29.953828  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
I20240503 02:12:29.970580  4193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e9898815c904092af0a6f238ed961bd"
dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33960
I20240503 02:12:29.970819  4193 raft_consensus.cc:595] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 2 LEADER]: Received request to transfer leadership
W20240503 02:12:29.978809  4152 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:30.007045  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
W20240503 02:12:30.032721  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
I20240503 02:12:30.052898  6539 raft_consensus.cc:979] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2: : Instructing follower 9c804fe67ee34e7fae5fab23db90f298 to start an election
I20240503 02:12:30.053069  6539 raft_consensus.cc:1067] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 2 LEADER]: Signalling peer 9c804fe67ee34e7fae5fab23db90f298 to start an election
I20240503 02:12:30.054405  5634 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e9898815c904092af0a6f238ed961bd"
dest_uuid: "9c804fe67ee34e7fae5fab23db90f298"
 from {username='slave'} at 127.0.145.2:52199
I20240503 02:12:30.054872  5634 raft_consensus.cc:482] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:30.055148  5634 raft_consensus.cc:3037] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:12:30.059561  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Service unavailable: leader transfer in progress
I20240503 02:12:30.061579  5634 raft_consensus.cc:504] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } }
I20240503 02:12:30.067103  4193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9898815c904092af0a6f238ed961bd" candidate_uuid: "9c804fe67ee34e7fae5fab23db90f298" candidate_term: 3 candidate_status { last_received { term: 2 index: 244 } } ignore_live_leader: true dest_uuid: "d7f19d54f56849f495dc547de6a47ad2"
I20240503 02:12:30.067365  4193 raft_consensus.cc:3032] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:12:30.067505  4193 raft_consensus.cc:729] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7f19d54f56849f495dc547de6a47ad2, State: Running, Role: LEADER
I20240503 02:12:30.067770  4193 consensus_queue.cc:260] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 2.244, Last appended by leader: 244, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } }
I20240503 02:12:30.068168  4193 raft_consensus.cc:3037] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:30.071427  4193 raft_consensus.cc:2454] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c804fe67ee34e7fae5fab23db90f298 in term 3.
I20240503 02:12:30.072700  5570 leader_election.cc:302] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c804fe67ee34e7fae5fab23db90f298, d7f19d54f56849f495dc547de6a47ad2; no voters: 
I20240503 02:12:30.084448  5634 leader_election.cc:288] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [CANDIDATE]: Term 3 election: Requested vote from peers d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433), 6ebeaff495e042c78067c5b18600dc98 (127.0.145.4:33659)
W20240503 02:12:30.087246  4154 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52786: Illegal state: replica d7f19d54f56849f495dc547de6a47ad2 is not leader of this config: current role FOLLOWER
I20240503 02:12:30.096349  6754 raft_consensus.cc:2781] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:12:30.112774  6754 raft_consensus.cc:686] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [term 3 LEADER]: Becoming Leader. State: Replica: 9c804fe67ee34e7fae5fab23db90f298, State: Running, Role: LEADER
I20240503 02:12:30.114105  5371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e9898815c904092af0a6f238ed961bd" candidate_uuid: "9c804fe67ee34e7fae5fab23db90f298" candidate_term: 3 candidate_status { last_received { term: 2 index: 244 } } ignore_live_leader: true dest_uuid: "6ebeaff495e042c78067c5b18600dc98"
I20240503 02:12:30.114382  5371 raft_consensus.cc:3037] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:30.116781  6754 consensus_queue.cc:237] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 244, Committed index: 244, Last appended: 2.244, Last appended by leader: 244, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } }
I20240503 02:12:30.120373  5371 raft_consensus.cc:2454] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c804fe67ee34e7fae5fab23db90f298 in term 3.
I20240503 02:12:30.121538  3915 catalog_manager.cc:5484] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 reported cstate change: term changed from 2 to 3, leader changed from d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) to 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6). New cstate: current_term: 3 leader_uuid: "9c804fe67ee34e7fae5fab23db90f298" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:30.177871  4193 raft_consensus.cc:1261] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 3 FOLLOWER]: Refusing update from remote peer 9c804fe67ee34e7fae5fab23db90f298: Log matching property violated. Preceding OpId in replica: term: 2 index: 244. Preceding OpId from leader: term: 3 index: 246. (index mismatch)
I20240503 02:12:30.179147  6754 consensus_queue.cc:1035] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20240503 02:12:30.179844  5374 raft_consensus.cc:1261] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 3 FOLLOWER]: Refusing update from remote peer 9c804fe67ee34e7fae5fab23db90f298: Log matching property violated. Preceding OpId in replica: term: 2 index: 244. Preceding OpId from leader: term: 3 index: 246. (index mismatch)
I20240503 02:12:30.182773  6755 consensus_queue.cc:1035] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 245, Last known committed idx: 244, Time since last communication: 0.000s
I20240503 02:12:30.184967  6696 mvcc.cc:205] Tried to move back new op lower bound from 7023420826316410880 to 7023420826085408768. Current Snapshot: MvccSnapshot[applied={T|T < 7023420826316410880 or (T in {7023420826316410880})}]
I20240503 02:12:30.307233  5631 consensus_queue.cc:237] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 3.247, Last appended by leader: 244, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } }
I20240503 02:12:30.317157  5374 raft_consensus.cc:1261] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 3 FOLLOWER]: Refusing update from remote peer 9c804fe67ee34e7fae5fab23db90f298: Log matching property violated. Preceding OpId in replica: term: 3 index: 247. Preceding OpId from leader: term: 3 index: 248. (index mismatch)
I20240503 02:12:30.317801  6755 consensus_queue.cc:1035] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240503 02:12:30.326490  6754 raft_consensus.cc:2932] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 [term 3 LEADER]: Committing config change with OpId 3.248: config changed from index 238 to 248, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:30.332103  5374 raft_consensus.cc:2932] T 6e9898815c904092af0a6f238ed961bd P 6ebeaff495e042c78067c5b18600dc98 [term 3 FOLLOWER]: Committing config change with OpId 3.248: config changed from index 238 to 248, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } } }
I20240503 02:12:30.335875  3908 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e9898815c904092af0a6f238ed961bd with cas_config_opid_index 238: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:30.337721  3920 catalog_manager.cc:5484] T 6e9898815c904092af0a6f238ed961bd P 9c804fe67ee34e7fae5fab23db90f298 reported cstate change: config changed from index 238 to 248, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New cstate: current_term: 3 leader_uuid: "9c804fe67ee34e7fae5fab23db90f298" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "6ebeaff495e042c78067c5b18600dc98" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 33659 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:30.342805  4173 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e9898815c904092af0a6f238ed961bd with delete_type TABLET_DATA_TOMBSTONED (TS d7f19d54f56849f495dc547de6a47ad2 not found in new config with opid_index 248) from {username='slave'} at 127.0.0.1:55800
I20240503 02:12:30.347072  6773 tablet_replica.cc:330] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2: stopping tablet replica
I20240503 02:12:30.347347  6773 raft_consensus.cc:2229] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:30.347640  6773 raft_consensus.cc:2258] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:30.348572  6773 ts_tablet_manager.cc:1875] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:30.354758  6773 ts_tablet_manager.cc:1888] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.247
I20240503 02:12:30.355031  6773 log.cc:1196] T 6e9898815c904092af0a6f238ed961bd P d7f19d54f56849f495dc547de6a47ad2: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/6e9898815c904092af0a6f238ed961bd
I20240503 02:12:30.357391  3908 catalog_manager.cc:4830] TS d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433): tablet 6e9898815c904092af0a6f238ed961bd (table rebalance_test_table_4 [id=7bd39e7281604b6ebd6334a68999befc]) successfully deleted
I20240503 02:12:30.939548  5485 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:30.978430  4305 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:31.067399  5615 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:31.101066  4043 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:31.119575  5745 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:31.120626  5354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:31.223335  4173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:32.207118  5485 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:32.454984  5615 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:32.491106  5745 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:32.494146  4305 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:32.524962  5354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:32.542217  4173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:32.640724  4043 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:34.730974  5485 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:34.765604  5354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:34.773232  5745 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:12:34.842581  4260 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.5:39125, user_credentials={real_user=slave}} blocked reactor thread for 88264.1us
I20240503 02:12:34.848788  5615 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:34.853224  4305 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:34.941907  4173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:35.026101  4043 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:36.158736  5485 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:36.284070  5745 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:36.374234  5354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:36.391738  4305 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:36.408455  4173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:36.408560  5615 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:36.424634  4043 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:37.500927  5485 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:37.510654  4305 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:37.512827  5354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:37.523064  5745 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:37.554222  5615 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:37.582649  4043 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:37.585978  4173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 1a0bcd52c0054aa6a0012cbd524d103c | 127.0.145.62:40001 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                               Value                                                                                |         Tags          |         Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                               | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                               | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                  | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                               | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                               | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                               | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                 | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.0.145.20:36429 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address      | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
 37c60eb66a6b45aaac4e0d7061572da6 | 127.0.145.5:39125 | HEALTHY | <none>   |       1        |       0
 6c3559ccff5d4c6681b3c90f892143dc | 127.0.145.7:35457 | HEALTHY | <none>   |       0        |       0
 6ebeaff495e042c78067c5b18600dc98 | 127.0.145.4:33659 | HEALTHY | <none>   |       0        |       0
 8b812251ceb44fbe8a54fea04a7f8dfc | 127.0.145.3:41599 | HEALTHY | <none>   |       5        |       0
 9c804fe67ee34e7fae5fab23db90f298 | 127.0.145.6:36423 | HEALTHY | <none>   |       1        |       0
 c4af56cee8e546c384cf20f40deeacd7 | 127.0.145.1:43835 | HEALTHY | <none>   |       5        |       0
 d7f19d54f56849f495dc547de6a47ad2 | 127.0.145.2:44433 | HEALTHY | <none>   |       4        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       7

Unusual flags for Tablet Server:
                   Flag                   |                                                                             Value                                                                              |         Tags          |      Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                            | experimental          | all 7 server(s) checked
 local_ip_for_outbound_sockets            | 127.0.145.1                                                                                                                                                    | experimental          | 127.0.145.1:43835
 local_ip_for_outbound_sockets            | 127.0.145.2                                                                                                                                                    | experimental          | 127.0.145.2:44433
 local_ip_for_outbound_sockets            | 127.0.145.3                                                                                                                                                    | experimental          | 127.0.145.3:41599
 local_ip_for_outbound_sockets            | 127.0.145.4                                                                                                                                                    | experimental          | 127.0.145.4:33659
 local_ip_for_outbound_sockets            | 127.0.145.5                                                                                                                                                    | experimental          | 127.0.145.5:39125
 local_ip_for_outbound_sockets            | 127.0.145.6                                                                                                                                                    | experimental          | 127.0.145.6:36423
 local_ip_for_outbound_sockets            | 127.0.145.7                                                                                                                                                    | experimental          | 127.0.145.7:35457
 never_fsync                              | true                                                                                                                                                           | unsafe,advanced       | all 7 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                              | unsafe,hidden         | all 7 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                           | experimental,advanced | all 7 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                           | unsafe                | all 7 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                             | hidden                | all 7 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.0.145.1:43835
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.0.145.2:44433
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.0.145.3:41599
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.0.145.4:33659
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.0.145.5:39125
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.0.145.6:36423
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.0.145.7:35457

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.0.145.20:36429 | all 7 server(s) checked
 time_source         | builtin            | all 7 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 8 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 5
 Third Quartile | 5
 Maximum        | 5

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 7
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240503 02:12:37.653745   580 log_verifier.cc:126] Checking tablet 07df29aeb41b427a8dcaf4a1a7db30da
I20240503 02:12:37.814112   580 log_verifier.cc:177] Verified matching terms for 421 ops in tablet 07df29aeb41b427a8dcaf4a1a7db30da
I20240503 02:12:37.814785   580 log_verifier.cc:126] Checking tablet 15fafdc84fec4b46979c311f377fd312
I20240503 02:12:37.982842   580 log_verifier.cc:177] Verified matching terms for 411 ops in tablet 15fafdc84fec4b46979c311f377fd312
I20240503 02:12:37.983440   580 log_verifier.cc:126] Checking tablet 1c1ff96ff85b4d5e9bd80fe5838b16db
I20240503 02:12:38.003518   580 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 1c1ff96ff85b4d5e9bd80fe5838b16db
I20240503 02:12:38.003665   580 log_verifier.cc:126] Checking tablet 27609e0e32a34efc915bbff6750f8234
I20240503 02:12:38.166076   580 log_verifier.cc:177] Verified matching terms for 385 ops in tablet 27609e0e32a34efc915bbff6750f8234
I20240503 02:12:38.166729   580 log_verifier.cc:126] Checking tablet 368fca99d20d4d6e826c45e44379c056
I20240503 02:12:38.290031   580 log_verifier.cc:177] Verified matching terms for 323 ops in tablet 368fca99d20d4d6e826c45e44379c056
I20240503 02:12:38.290501   580 log_verifier.cc:126] Checking tablet 6e9898815c904092af0a6f238ed961bd
I20240503 02:12:38.437534   580 log_verifier.cc:177] Verified matching terms for 379 ops in tablet 6e9898815c904092af0a6f238ed961bd
I20240503 02:12:38.438036   580 log_verifier.cc:126] Checking tablet 75a2da7ca68b43e28cad0847e4d27982
I20240503 02:12:38.596789   580 log_verifier.cc:177] Verified matching terms for 428 ops in tablet 75a2da7ca68b43e28cad0847e4d27982
I20240503 02:12:38.597298   580 log_verifier.cc:126] Checking tablet 8334f743021f4957a427800af656bec1
I20240503 02:12:38.785516   580 log_verifier.cc:177] Verified matching terms for 416 ops in tablet 8334f743021f4957a427800af656bec1
I20240503 02:12:38.786069   580 log_verifier.cc:126] Checking tablet 87e472f1975b4141b795aa6c3b75c7c0
I20240503 02:12:38.958930   580 log_verifier.cc:177] Verified matching terms for 429 ops in tablet 87e472f1975b4141b795aa6c3b75c7c0
I20240503 02:12:38.959475   580 log_verifier.cc:126] Checking tablet 8b76c95bac0f497f8ab21c5295028026
I20240503 02:12:39.122275   580 log_verifier.cc:177] Verified matching terms for 390 ops in tablet 8b76c95bac0f497f8ab21c5295028026
I20240503 02:12:39.122888   580 log_verifier.cc:126] Checking tablet ccccc26d94514b87a91b0c00fa18e669
I20240503 02:12:39.268150   580 log_verifier.cc:177] Verified matching terms for 390 ops in tablet ccccc26d94514b87a91b0c00fa18e669
I20240503 02:12:39.268744   580 log_verifier.cc:126] Checking tablet d846ad6206514c79b142064043ad1cb0
I20240503 02:12:39.400161   580 log_verifier.cc:177] Verified matching terms for 333 ops in tablet d846ad6206514c79b142064043ad1cb0
I20240503 02:12:39.400597   580 log_verifier.cc:126] Checking tablet d98afd14a7f14415a0f404dea8750d82
I20240503 02:12:39.549997   580 log_verifier.cc:177] Verified matching terms for 393 ops in tablet d98afd14a7f14415a0f404dea8750d82
I20240503 02:12:39.550495   580 log_verifier.cc:126] Checking tablet df156172a4554094b36cd5cc3876c922
I20240503 02:12:39.699534   580 log_verifier.cc:177] Verified matching terms for 381 ops in tablet df156172a4554094b36cd5cc3876c922
I20240503 02:12:39.700021   580 log_verifier.cc:126] Checking tablet ec5c9e506c2f4c29aea9a9de259a0c19
I20240503 02:12:39.828524   580 log_verifier.cc:177] Verified matching terms for 332 ops in tablet ec5c9e506c2f4c29aea9a9de259a0c19
I20240503 02:12:39.828967   580 log_verifier.cc:126] Checking tablet ed79dde0a1f4419c8085c042ed5b8780
I20240503 02:12:39.968952   580 log_verifier.cc:177] Verified matching terms for 352 ops in tablet ed79dde0a1f4419c8085c042ed5b8780
I20240503 02:12:39.973906   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.8:0
--local_ip_for_outbound_sockets=127.0.145.8
--webserver_interface=127.0.145.8
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--builtin_ntp_servers=127.0.145.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:12:40.484346  7075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:40.484673  7075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:40.484843  7075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:40.492707  7075 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:40.492884  7075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:40.493053  7075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.8
I20240503 02:12:40.504156  7075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.145.8
--webserver_port=0
--tserver_master_addrs=127.0.145.62:40001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.8
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:40.505558  7075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:40.507514  7075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:40.522140  7081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:40.523077  7082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:40.523921  7084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:40.524776  7075 server_base.cc:1034] running on GCE node
I20240503 02:12:40.525445  7075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:40.526469  7075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:40.527688  7075 hybrid_clock.cc:648] HybridClock initialized: now 1714702360527641 us; error 47 us; skew 500 ppm
I20240503 02:12:40.528039  7075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:40.530740  7075 webserver.cc:460] Webserver started at http://127.0.145.8:38449/ using document root <none> and password file <none>
I20240503 02:12:40.531535  7075 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:40.531678  7075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:40.532013  7075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:40.534808  7075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "207a8023eda24dc89a2e95241d94f89a"
format_stamp: "Formatted at 2024-05-03 02:12:40 on dist-test-slave-gq9h"
I20240503 02:12:40.535648  7075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "207a8023eda24dc89a2e95241d94f89a"
format_stamp: "Formatted at 2024-05-03 02:12:40 on dist-test-slave-gq9h"
I20240503 02:12:40.542304  7075 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20240503 02:12:40.546573  7090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:40.548070  7075 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:12:40.548229  7075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "207a8023eda24dc89a2e95241d94f89a"
format_stamp: "Formatted at 2024-05-03 02:12:40 on dist-test-slave-gq9h"
I20240503 02:12:40.548429  7075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:40.592615  7075 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:40.593706  7075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:40.594061  7075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:40.595342  7075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:40.597077  7075 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:40.597200  7075 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:40.597322  7075 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:40.597407  7075 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:40.641121  7075 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.8:34161
I20240503 02:12:40.641281  7202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.8:34161 every 8 connection(s)
I20240503 02:12:40.642630  7075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:12:40.651736   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 7075
I20240503 02:12:40.651947   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:12:40.657457  7203 heartbeater.cc:347] Connected to a master server at 127.0.145.62:40001
I20240503 02:12:40.657792  7203 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:40.658757  7203 heartbeater.cc:511] Master 127.0.145.62:40001 requested a full tablet report, sending...
I20240503 02:12:40.660799  3920 ts_manager.cc:194] Registered new tserver with Master: 207a8023eda24dc89a2e95241d94f89a (127.0.145.8:34161)
I20240503 02:12:40.662015  3920 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.8:50035
I20240503 02:12:41.209789  5485 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:41.211942  5745 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:41.212680  4305 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:41.213876  7138 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:41.231995  4043 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:41.247148  5354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:41.251345  5615 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:41.267908  4173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:41.471648  7138 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:41.487921  5745 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:41.490782  5485 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:41.503798  4043 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:41.507208  5615 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:41.509274  4173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:41.511814  5354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:41.520823  4305 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:41.644577  4322 consensus_queue.cc:237] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 385, Committed index: 385, Last appended: 1.385, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } }
I20240503 02:12:41.647789  5504 raft_consensus.cc:1261] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 1 index: 386. (index mismatch)
I20240503 02:12:41.648559  7300 consensus_queue.cc:1035] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s
W20240503 02:12:41.654918  4259 consensus_peers.cc:485] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc -> Peer 207a8023eda24dc89a2e95241d94f89a (127.0.145.8:34161): Couldn't send request to peer 207a8023eda24dc89a2e95241d94f89a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27609e0e32a34efc915bbff6750f8234. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:41.659765  6545 raft_consensus.cc:2932] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.386: config changed from index 109 to 386, NON_VOTER 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) added. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } } }
I20240503 02:12:41.660948  5504 raft_consensus.cc:2932] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Committing config change with OpId 1.386: config changed from index 109 to 386, NON_VOTER 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) added. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } } }
I20240503 02:12:41.664563  3921 catalog_manager.cc:5484] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index 109 to 386, NON_VOTER 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:41.665022  7203 heartbeater.cc:503] Master 127.0.145.62:40001 was elected leader, sending a full tablet report...
I20240503 02:12:41.679358  4062 consensus_queue.cc:237] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 390, Committed index: 390, Last appended: 1.390, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } }
I20240503 02:12:41.682165  5504 raft_consensus.cc:1261] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 390. Preceding OpId from leader: term: 1 index: 391. (index mismatch)
I20240503 02:12:41.683219  7313 consensus_queue.cc:1035] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391, Last known committed idx: 390, Time since last communication: 0.000s
I20240503 02:12:41.686574  6963 raft_consensus.cc:2932] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.391: config changed from index 100 to 391, NON_VOTER 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) added. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } } }
I20240503 02:12:41.688624  5504 raft_consensus.cc:2932] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Committing config change with OpId 1.391: config changed from index 100 to 391, NON_VOTER 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) added. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } } }
I20240503 02:12:41.692737  3915 catalog_manager.cc:5484] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index 100 to 391, NON_VOTER 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:41.694252  3997 consensus_peers.cc:485] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 -> Peer 207a8023eda24dc89a2e95241d94f89a (127.0.145.8:34161): Couldn't send request to peer 207a8023eda24dc89a2e95241d94f89a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b76c95bac0f497f8ab21c5295028026. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:41.706782  5504 consensus_queue.cc:237] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } }
I20240503 02:12:41.709702  4193 raft_consensus.cc:1261] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Refusing update from remote peer 37c60eb66a6b45aaac4e0d7061572da6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:41.710412  7282 consensus_queue.cc:1035] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:41.714666  4193 raft_consensus.cc:2932] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } } }
I20240503 02:12:41.713747  7322 raft_consensus.cc:2932] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } } }
W20240503 02:12:41.717622  5439 consensus_peers.cc:485] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 -> Peer 207a8023eda24dc89a2e95241d94f89a (127.0.145.8:34161): Couldn't send request to peer 207a8023eda24dc89a2e95241d94f89a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c1ff96ff85b4d5e9bd80fe5838b16db. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:41.720316  3921 catalog_manager.cc:5484] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: config changed from index 5 to 6, NON_VOTER 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) added. New cstate: current_term: 2 leader_uuid: "37c60eb66a6b45aaac4e0d7061572da6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } } }
I20240503 02:12:41.734846  4062 consensus_queue.cc:237] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 333, Committed index: 333, Last appended: 1.333, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false replace: true } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } }
I20240503 02:12:41.737057  5631 raft_consensus.cc:1261] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 333. Preceding OpId from leader: term: 1 index: 334. (index mismatch)
W20240503 02:12:41.737344  3997 consensus_peers.cc:485] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 -> Peer 207a8023eda24dc89a2e95241d94f89a (127.0.145.8:34161): Couldn't send request to peer 207a8023eda24dc89a2e95241d94f89a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d846ad6206514c79b142064043ad1cb0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:41.737681  7313 consensus_queue.cc:1035] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I20240503 02:12:41.740617  6963 raft_consensus.cc:2932] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.334: config changed from index 41 to 334, NON_VOTER 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false replace: true } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } } }
I20240503 02:12:41.741276  5631 raft_consensus.cc:2932] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Committing config change with OpId 1.334: config changed from index 41 to 334, NON_VOTER 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) added. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false replace: true } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } } }
I20240503 02:12:41.746275  3915 catalog_manager.cc:5484] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index 41 to 334, NON_VOTER 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:42.192591  7374 ts_tablet_manager.cc:902] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a: Initiating tablet copy from peer c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:42.192787  7375 ts_tablet_manager.cc:902] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a: Initiating tablet copy from peer 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5:39125)
I20240503 02:12:42.199534  7374 tablet_copy_client.cc:308] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:43835
I20240503 02:12:42.199524  7375 tablet_copy_client.cc:308] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.5:39125
I20240503 02:12:42.209174  4083 tablet_copy_service.cc:140] P c4af56cee8e546c384cf20f40deeacd7: Received BeginTabletCopySession request for tablet 8b76c95bac0f497f8ab21c5295028026 from peer 207a8023eda24dc89a2e95241d94f89a ({username='slave'} at 127.0.145.8:45271)
I20240503 02:12:42.209249  5525 tablet_copy_service.cc:140] P 37c60eb66a6b45aaac4e0d7061572da6: Received BeginTabletCopySession request for tablet 1c1ff96ff85b4d5e9bd80fe5838b16db from peer 207a8023eda24dc89a2e95241d94f89a ({username='slave'} at 127.0.145.8:41019)
I20240503 02:12:42.209414  4083 tablet_copy_service.cc:161] P c4af56cee8e546c384cf20f40deeacd7: Beginning new tablet copy session on tablet 8b76c95bac0f497f8ab21c5295028026 from peer 207a8023eda24dc89a2e95241d94f89a at {username='slave'} at 127.0.145.8:45271: session id = 207a8023eda24dc89a2e95241d94f89a-8b76c95bac0f497f8ab21c5295028026
I20240503 02:12:42.209465  5525 tablet_copy_service.cc:161] P 37c60eb66a6b45aaac4e0d7061572da6: Beginning new tablet copy session on tablet 1c1ff96ff85b4d5e9bd80fe5838b16db from peer 207a8023eda24dc89a2e95241d94f89a at {username='slave'} at 127.0.145.8:41019: session id = 207a8023eda24dc89a2e95241d94f89a-1c1ff96ff85b4d5e9bd80fe5838b16db
I20240503 02:12:42.211848  7386 ts_tablet_manager.cc:902] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a: Initiating tablet copy from peer 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599)
I20240503 02:12:42.212529  4083 tablet_copy_source_session.cc:205] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:42.213625  5525 tablet_copy_source_session.cc:205] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:42.216542  7386 tablet_copy_client.cc:308] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:41599
I20240503 02:12:42.219360  7374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b76c95bac0f497f8ab21c5295028026. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.219956  7375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c1ff96ff85b4d5e9bd80fe5838b16db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.223866  4345 tablet_copy_service.cc:140] P 8b812251ceb44fbe8a54fea04a7f8dfc: Received BeginTabletCopySession request for tablet 27609e0e32a34efc915bbff6750f8234 from peer 207a8023eda24dc89a2e95241d94f89a ({username='slave'} at 127.0.145.8:35457)
I20240503 02:12:42.224093  4345 tablet_copy_service.cc:161] P 8b812251ceb44fbe8a54fea04a7f8dfc: Beginning new tablet copy session on tablet 27609e0e32a34efc915bbff6750f8234 from peer 207a8023eda24dc89a2e95241d94f89a at {username='slave'} at 127.0.145.8:35457: session id = 207a8023eda24dc89a2e95241d94f89a-27609e0e32a34efc915bbff6750f8234
I20240503 02:12:42.227219  4345 tablet_copy_source_session.cc:205] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:42.228844  7386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27609e0e32a34efc915bbff6750f8234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.231848  7374 tablet_copy_client.cc:783] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:42.231913  7375 tablet_copy_client.cc:783] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:42.232234  7374 tablet_copy_client.cc:647] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:42.232398  7375 tablet_copy_client.cc:647] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:42.234858  7375 tablet_copy_client.cc:523] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:42.236310  7386 tablet_copy_client.cc:783] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:42.236567  7374 tablet_copy_client.cc:523] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:42.236677  7386 tablet_copy_client.cc:647] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:42.239974  7386 tablet_copy_client.cc:523] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:42.239974  7375 tablet_bootstrap.cc:492] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a: Bootstrap starting.
I20240503 02:12:42.243139  7374 tablet_bootstrap.cc:492] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a: Bootstrap starting.
I20240503 02:12:42.244462  7386 tablet_bootstrap.cc:492] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a: Bootstrap starting.
I20240503 02:12:42.253052  7375 log.cc:824] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:42.267694  7375 tablet_bootstrap.cc:492] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:42.268393  7375 tablet_bootstrap.cc:492] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a: Bootstrap complete.
I20240503 02:12:42.269001  7375 ts_tablet_manager.cc:1367] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a: Time spent bootstrapping tablet: real 0.029s	user 0.022s	sys 0.004s
I20240503 02:12:42.280074  7375 raft_consensus.cc:348] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } }
I20240503 02:12:42.280869  7375 raft_consensus.cc:729] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 207a8023eda24dc89a2e95241d94f89a, State: Initialized, Role: LEARNER
I20240503 02:12:42.281596  7375 consensus_queue.cc:260] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } }
I20240503 02:12:42.290853  7375 ts_tablet_manager.cc:1398] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.015s
I20240503 02:12:42.298650  5525 tablet_copy_service.cc:342] P 37c60eb66a6b45aaac4e0d7061572da6: Request end of tablet copy session 207a8023eda24dc89a2e95241d94f89a-1c1ff96ff85b4d5e9bd80fe5838b16db received from {username='slave'} at 127.0.145.8:41019
I20240503 02:12:42.298861  5525 tablet_copy_service.cc:433] P 37c60eb66a6b45aaac4e0d7061572da6: ending tablet copy session 207a8023eda24dc89a2e95241d94f89a-1c1ff96ff85b4d5e9bd80fe5838b16db on tablet 1c1ff96ff85b4d5e9bd80fe5838b16db with peer 207a8023eda24dc89a2e95241d94f89a
I20240503 02:12:42.306268  7375 ts_tablet_manager.cc:902] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a: Initiating tablet copy from peer c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
I20240503 02:12:42.318190  7375 tablet_copy_client.cc:308] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:43835
I20240503 02:12:42.319377  4083 tablet_copy_service.cc:140] P c4af56cee8e546c384cf20f40deeacd7: Received BeginTabletCopySession request for tablet d846ad6206514c79b142064043ad1cb0 from peer 207a8023eda24dc89a2e95241d94f89a ({username='slave'} at 127.0.145.8:45271)
I20240503 02:12:42.319567  4083 tablet_copy_service.cc:161] P c4af56cee8e546c384cf20f40deeacd7: Beginning new tablet copy session on tablet d846ad6206514c79b142064043ad1cb0 from peer 207a8023eda24dc89a2e95241d94f89a at {username='slave'} at 127.0.145.8:45271: session id = 207a8023eda24dc89a2e95241d94f89a-d846ad6206514c79b142064043ad1cb0
I20240503 02:12:42.322789  4083 tablet_copy_source_session.cc:205] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:42.324469  7375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d846ad6206514c79b142064043ad1cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.332633  7375 tablet_copy_client.cc:783] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:42.332995  7375 tablet_copy_client.cc:647] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:42.335860  7375 tablet_copy_client.cc:523] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:42.344995  7375 tablet_bootstrap.cc:492] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a: Bootstrap starting.
I20240503 02:12:42.774190  7158 raft_consensus.cc:1203] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6]   Dedup: 2.6->[]
I20240503 02:12:42.823521  7386 tablet_bootstrap.cc:492] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a: Bootstrap replayed 1/1 log segments. Stats: ops{read=386 overwritten=0 applied=386 ignored=0} inserts{seen=1014 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:42.826292  7386 tablet_bootstrap.cc:492] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a: Bootstrap complete.
I20240503 02:12:42.826781  7386 ts_tablet_manager.cc:1367] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a: Time spent bootstrapping tablet: real 0.582s	user 0.444s	sys 0.061s
I20240503 02:12:42.828011  7386 raft_consensus.cc:348] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } }
I20240503 02:12:42.828660  7386 raft_consensus.cc:729] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 207a8023eda24dc89a2e95241d94f89a, State: Initialized, Role: LEARNER
I20240503 02:12:42.829327  7386 consensus_queue.cc:260] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 386, Last appended: 1.386, Last appended by leader: 386, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } }
I20240503 02:12:42.830029  7386 ts_tablet_manager.cc:1398] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 02:12:42.832654  7374 tablet_bootstrap.cc:492] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a: Bootstrap replayed 1/1 log segments. Stats: ops{read=391 overwritten=0 applied=391 ignored=0} inserts{seen=1060 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:42.833164  7374 tablet_bootstrap.cc:492] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a: Bootstrap complete.
I20240503 02:12:42.833514  7374 ts_tablet_manager.cc:1367] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a: Time spent bootstrapping tablet: real 0.591s	user 0.462s	sys 0.039s
I20240503 02:12:42.833731  4345 tablet_copy_service.cc:342] P 8b812251ceb44fbe8a54fea04a7f8dfc: Request end of tablet copy session 207a8023eda24dc89a2e95241d94f89a-27609e0e32a34efc915bbff6750f8234 received from {username='slave'} at 127.0.145.8:35457
I20240503 02:12:42.833854  4345 tablet_copy_service.cc:433] P 8b812251ceb44fbe8a54fea04a7f8dfc: ending tablet copy session 207a8023eda24dc89a2e95241d94f89a-27609e0e32a34efc915bbff6750f8234 on tablet 27609e0e32a34efc915bbff6750f8234 with peer 207a8023eda24dc89a2e95241d94f89a
I20240503 02:12:42.834373  7374 raft_consensus.cc:348] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } }
I20240503 02:12:42.834718  7374 raft_consensus.cc:729] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 207a8023eda24dc89a2e95241d94f89a, State: Initialized, Role: LEARNER
I20240503 02:12:42.834939  7374 consensus_queue.cc:260] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 391, Last appended: 1.391, Last appended by leader: 391, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } }
I20240503 02:12:42.835609  7374 ts_tablet_manager.cc:1398] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 02:12:42.836953  4083 tablet_copy_service.cc:342] P c4af56cee8e546c384cf20f40deeacd7: Request end of tablet copy session 207a8023eda24dc89a2e95241d94f89a-8b76c95bac0f497f8ab21c5295028026 received from {username='slave'} at 127.0.145.8:45271
I20240503 02:12:42.837118  4083 tablet_copy_service.cc:433] P c4af56cee8e546c384cf20f40deeacd7: ending tablet copy session 207a8023eda24dc89a2e95241d94f89a-8b76c95bac0f497f8ab21c5295028026 on tablet 8b76c95bac0f497f8ab21c5295028026 with peer 207a8023eda24dc89a2e95241d94f89a
I20240503 02:12:42.872308  7375 tablet_bootstrap.cc:492] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a: Bootstrap replayed 1/1 log segments. Stats: ops{read=334 overwritten=0 applied=334 ignored=0} inserts{seen=1003 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:42.872756  7375 tablet_bootstrap.cc:492] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a: Bootstrap complete.
I20240503 02:12:42.873046  7375 ts_tablet_manager.cc:1367] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a: Time spent bootstrapping tablet: real 0.528s	user 0.425s	sys 0.028s
I20240503 02:12:42.873766  7375 raft_consensus.cc:348] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false replace: true } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } }
I20240503 02:12:42.873962  7375 raft_consensus.cc:729] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 207a8023eda24dc89a2e95241d94f89a, State: Initialized, Role: LEARNER
I20240503 02:12:42.874191  7375 consensus_queue.cc:260] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 334, Last appended: 1.334, Last appended by leader: 334, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false replace: true } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: true } }
I20240503 02:12:42.874760  7375 ts_tablet_manager.cc:1398] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 02:12:42.875946  4083 tablet_copy_service.cc:342] P c4af56cee8e546c384cf20f40deeacd7: Request end of tablet copy session 207a8023eda24dc89a2e95241d94f89a-d846ad6206514c79b142064043ad1cb0 received from {username='slave'} at 127.0.145.8:45271
I20240503 02:12:42.876618  4083 tablet_copy_service.cc:433] P c4af56cee8e546c384cf20f40deeacd7: ending tablet copy session 207a8023eda24dc89a2e95241d94f89a-d846ad6206514c79b142064043ad1cb0 on tablet d846ad6206514c79b142064043ad1cb0 with peer 207a8023eda24dc89a2e95241d94f89a
I20240503 02:12:42.896133  7158 raft_consensus.cc:1203] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.333->[1.334-1.334]   Dedup: 1.334->[]
I20240503 02:12:43.220234  7158 raft_consensus.cc:1203] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.390->[1.391-1.391]   Dedup: 1.391->[]
I20240503 02:12:43.259234  7435 raft_consensus.cc:1050] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6: attempting to promote NON_VOTER 207a8023eda24dc89a2e95241d94f89a to VOTER
I20240503 02:12:43.260293  7435 consensus_queue.cc:237] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:43.263502  4193 raft_consensus.cc:1261] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Refusing update from remote peer 37c60eb66a6b45aaac4e0d7061572da6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:12:43.263584  7158 raft_consensus.cc:1261] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a [term 2 LEARNER]: Refusing update from remote peer 37c60eb66a6b45aaac4e0d7061572da6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:12:43.264181  7466 consensus_queue.cc:1035] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:12:43.264539  7435 consensus_queue.cc:1035] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:12:43.268404  7466 raft_consensus.cc:2932] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.269210  4193 raft_consensus.cc:2932] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.271384  7158 raft_consensus.cc:2932] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.275215  3915 catalog_manager.cc:5484] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 reported cstate change: config changed from index 6 to 7, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "37c60eb66a6b45aaac4e0d7061572da6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d7f19d54f56849f495dc547de6a47ad2" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 44433 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.283666  5504 consensus_queue.cc:237] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:43.285986  7158 raft_consensus.cc:1261] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a [term 2 FOLLOWER]: Refusing update from remote peer 37c60eb66a6b45aaac4e0d7061572da6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:12:43.286672  7466 consensus_queue.cc:1035] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:12:43.289937  7435 raft_consensus.cc:2932] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.293696  3908 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c1ff96ff85b4d5e9bd80fe5838b16db with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:43.294061  7158 raft_consensus.cc:2932] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 207a8023eda24dc89a2e95241d94f89a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.295213  3919 catalog_manager.cc:5484] T 1c1ff96ff85b4d5e9bd80fe5838b16db P 37c60eb66a6b45aaac4e0d7061572da6 reported cstate change: config changed from index 7 to 8, VOTER d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2) evicted. New cstate: current_term: 2 leader_uuid: "37c60eb66a6b45aaac4e0d7061572da6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:43.300565  4173 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c1ff96ff85b4d5e9bd80fe5838b16db with delete_type TABLET_DATA_TOMBSTONED (TS d7f19d54f56849f495dc547de6a47ad2 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:55800
I20240503 02:12:43.303082  7476 tablet_replica.cc:330] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2: stopping tablet replica
I20240503 02:12:43.303292  7476 raft_consensus.cc:2229] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.303557  7476 raft_consensus.cc:2258] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.305544  7476 ts_tablet_manager.cc:1875] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:43.311801  7476 ts_tablet_manager.cc:1888] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 02:12:43.311960  7476 log.cc:1196] T 1c1ff96ff85b4d5e9bd80fe5838b16db P d7f19d54f56849f495dc547de6a47ad2: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/1c1ff96ff85b4d5e9bd80fe5838b16db
I20240503 02:12:43.312803  3908 catalog_manager.cc:4830] TS d7f19d54f56849f495dc547de6a47ad2 (127.0.145.2:44433): tablet 1c1ff96ff85b4d5e9bd80fe5838b16db (table TestTable [id=a2f222c7915242918dc88c24b67a23ef]) successfully deleted
I20240503 02:12:43.344548  7444 raft_consensus.cc:1050] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7: attempting to promote NON_VOTER 207a8023eda24dc89a2e95241d94f89a to VOTER
I20240503 02:12:43.345358  7444 consensus_queue.cc:237] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 334, Committed index: 334, Last appended: 1.334, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false replace: true } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:43.348078  5631 raft_consensus.cc:1261] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 334. Preceding OpId from leader: term: 1 index: 335. (index mismatch)
I20240503 02:12:43.348337  7158 raft_consensus.cc:1261] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a [term 1 LEARNER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 334. Preceding OpId from leader: term: 1 index: 335. (index mismatch)
I20240503 02:12:43.348950  7433 consensus_queue.cc:1035] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.000s
I20240503 02:12:43.349419  7444 consensus_queue.cc:1035] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.000s
I20240503 02:12:43.352607  7313 raft_consensus.cc:2932] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.335: config changed from index 334 to 335, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false replace: true } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.353335  5631 raft_consensus.cc:2932] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Committing config change with OpId 1.335: config changed from index 334 to 335, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false replace: true } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.353487  7158 raft_consensus.cc:2932] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a [term 1 FOLLOWER]: Committing config change with OpId 1.335: config changed from index 334 to 335, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false replace: true } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.358657  3921 catalog_manager.cc:5484] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index 334 to 335, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c804fe67ee34e7fae5fab23db90f298" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 36423 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:43.364073  4062 consensus_queue.cc:237] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 1.335, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:43.366425  7158 raft_consensus.cc:1261] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 335. Preceding OpId from leader: term: 1 index: 336. (index mismatch)
I20240503 02:12:43.367236  7433 consensus_queue.cc:1035] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s
I20240503 02:12:43.370033  7444 raft_consensus.cc:2932] T d846ad6206514c79b142064043ad1cb0 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.336: config changed from index 335 to 336, VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) evicted. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.370889  7158 raft_consensus.cc:2932] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a [term 1 FOLLOWER]: Committing config change with OpId 1.336: config changed from index 335 to 336, VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) evicted. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.374559  3909 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d846ad6206514c79b142064043ad1cb0 with cas_config_opid_index 335: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:43.376341  3921 catalog_manager.cc:5484] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a reported cstate change: config changed from index 335 to 336, VOTER 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6) evicted. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.379757  7158 raft_consensus.cc:1203] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.385->[1.386-1.386]   Dedup: 1.386->[]
I20240503 02:12:43.383330  5615 tablet_service.cc:1514] Processing DeleteTablet for tablet d846ad6206514c79b142064043ad1cb0 with delete_type TABLET_DATA_TOMBSTONED (TS 9c804fe67ee34e7fae5fab23db90f298 not found in new config with opid_index 336) from {username='slave'} at 127.0.0.1:39972
I20240503 02:12:43.384429  7509 tablet_replica.cc:330] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: stopping tablet replica
I20240503 02:12:43.385185  7509 raft_consensus.cc:2229] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:43.385735  7509 raft_consensus.cc:2258] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:43.389182  7509 ts_tablet_manager.cc:1875] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:43.396090  7509 ts_tablet_manager.cc:1888] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.335
I20240503 02:12:43.396334  7509 log.cc:1196] T d846ad6206514c79b142064043ad1cb0 P 9c804fe67ee34e7fae5fab23db90f298: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal/wals/d846ad6206514c79b142064043ad1cb0
I20240503 02:12:43.397254  3908 catalog_manager.cc:4830] TS 9c804fe67ee34e7fae5fab23db90f298 (127.0.145.6:36423): tablet d846ad6206514c79b142064043ad1cb0 (table rebalance_test_table_2 [id=46a3ef9b51d44a84a25ae9ff7198c9b5]) successfully deleted
I20240503 02:12:43.812011  7444 raft_consensus.cc:1050] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7: attempting to promote NON_VOTER 207a8023eda24dc89a2e95241d94f89a to VOTER
I20240503 02:12:43.812664  7444 consensus_queue.cc:237] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 391, Committed index: 391, Last appended: 1.391, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:43.815131  7158 raft_consensus.cc:1261] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [term 1 LEARNER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 391. Preceding OpId from leader: term: 1 index: 392. (index mismatch)
I20240503 02:12:43.815116  5504 raft_consensus.cc:1261] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Refusing update from remote peer c4af56cee8e546c384cf20f40deeacd7: Log matching property violated. Preceding OpId in replica: term: 1 index: 391. Preceding OpId from leader: term: 1 index: 392. (index mismatch)
I20240503 02:12:43.815753  7313 consensus_queue.cc:1035] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 392, Last known committed idx: 391, Time since last communication: 0.000s
I20240503 02:12:43.816085  7444 consensus_queue.cc:1035] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 392, Last known committed idx: 391, Time since last communication: 0.000s
I20240503 02:12:43.819041  7433 raft_consensus.cc:2932] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Committing config change with OpId 1.392: config changed from index 391 to 392, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.819743  5504 raft_consensus.cc:2932] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Committing config change with OpId 1.392: config changed from index 391 to 392, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.823904  7158 raft_consensus.cc:2932] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [term 1 FOLLOWER]: Committing config change with OpId 1.392: config changed from index 391 to 392, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.832194  3921 catalog_manager.cc:5484] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 reported cstate change: config changed from index 391 to 392, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c4af56cee8e546c384cf20f40deeacd7" committed_config { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:43.836097  7431 raft_consensus.cc:1050] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc: attempting to promote NON_VOTER 207a8023eda24dc89a2e95241d94f89a to VOTER
I20240503 02:12:43.837100  7431 consensus_queue.cc:237] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 386, Committed index: 386, Last appended: 1.386, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:43.839572  5504 raft_consensus.cc:1261] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 1 index: 387. (index mismatch)
I20240503 02:12:43.839916  7158 raft_consensus.cc:1261] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a [term 1 LEARNER]: Refusing update from remote peer 8b812251ceb44fbe8a54fea04a7f8dfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 1 index: 387. (index mismatch)
I20240503 02:12:43.840205  7431 consensus_queue.cc:1035] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20240503 02:12:43.840601  7543 consensus_queue.cc:1035] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20240503 02:12:43.843694  7543 raft_consensus.cc:2932] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Committing config change with OpId 1.387: config changed from index 386 to 387, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.844949  5504 raft_consensus.cc:2932] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Committing config change with OpId 1.387: config changed from index 386 to 387, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.844918  7158 raft_consensus.cc:2932] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a [term 1 FOLLOWER]: Committing config change with OpId 1.387: config changed from index 386 to 387, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:43.848752  3915 catalog_manager.cc:5484] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc reported cstate change: config changed from index 386 to 387, 207a8023eda24dc89a2e95241d94f89a (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" committed_config { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:43.889262  4322 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "27609e0e32a34efc915bbff6750f8234"
dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54128
I20240503 02:12:43.889480  4322 raft_consensus.cc:595] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:43.910750  4062 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8b76c95bac0f497f8ab21c5295028026"
dest_uuid: "c4af56cee8e546c384cf20f40deeacd7"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41894
I20240503 02:12:43.910969  4062 raft_consensus.cc:595] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:44.130108  4322 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "27609e0e32a34efc915bbff6750f8234"
dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54148
I20240503 02:12:44.130354  4322 raft_consensus.cc:595] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:44.149747  4062 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8b76c95bac0f497f8ab21c5295028026"
dest_uuid: "c4af56cee8e546c384cf20f40deeacd7"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41918
I20240503 02:12:44.149966  4062 raft_consensus.cc:595] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:44.245736  7433 raft_consensus.cc:979] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7: : Instructing follower 207a8023eda24dc89a2e95241d94f89a to start an election
I20240503 02:12:44.245946  7433 raft_consensus.cc:1067] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Signalling peer 207a8023eda24dc89a2e95241d94f89a to start an election
I20240503 02:12:44.246785  7158 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8b76c95bac0f497f8ab21c5295028026"
dest_uuid: "207a8023eda24dc89a2e95241d94f89a"
 from {username='slave'} at 127.0.145.1:48241
I20240503 02:12:44.247092  7158 raft_consensus.cc:482] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:44.247282  7158 raft_consensus.cc:3037] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:44.249624  7158 raft_consensus.cc:504] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:44.251470  7543 raft_consensus.cc:979] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc: : Instructing follower 37c60eb66a6b45aaac4e0d7061572da6 to start an election
I20240503 02:12:44.251654  7543 raft_consensus.cc:1067] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Signalling peer 37c60eb66a6b45aaac4e0d7061572da6 to start an election
I20240503 02:12:44.251533  7158 leader_election.cc:288] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [CANDIDATE]: Term 2 election: Requested vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835), 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5:39125)
I20240503 02:12:44.252107  4062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b76c95bac0f497f8ab21c5295028026" candidate_uuid: "207a8023eda24dc89a2e95241d94f89a" candidate_term: 2 candidate_status { last_received { term: 1 index: 392 } } ignore_live_leader: true dest_uuid: "c4af56cee8e546c384cf20f40deeacd7"
I20240503 02:12:44.252126  5504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b76c95bac0f497f8ab21c5295028026" candidate_uuid: "207a8023eda24dc89a2e95241d94f89a" candidate_term: 2 candidate_status { last_received { term: 1 index: 392 } } ignore_live_leader: true dest_uuid: "37c60eb66a6b45aaac4e0d7061572da6"
I20240503 02:12:44.252350  4062 raft_consensus.cc:3032] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:44.252480  4062 raft_consensus.cc:729] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c4af56cee8e546c384cf20f40deeacd7, State: Running, Role: LEADER
I20240503 02:12:44.252475  5504 raft_consensus.cc:3037] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:44.252526  5505 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "27609e0e32a34efc915bbff6750f8234"
dest_uuid: "37c60eb66a6b45aaac4e0d7061572da6"
 from {username='slave'} at 127.0.145.3:52501
I20240503 02:12:44.252701  5505 raft_consensus.cc:482] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:44.252828  5505 raft_consensus.cc:3037] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:44.252813  4062 consensus_queue.cc:260] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 392, Committed index: 392, Last appended: 1.392, Last appended by leader: 392, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:44.253239  4062 raft_consensus.cc:3037] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:44.255681  4062 raft_consensus.cc:2454] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 207a8023eda24dc89a2e95241d94f89a in term 2.
I20240503 02:12:44.255760  5504 raft_consensus.cc:2454] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 207a8023eda24dc89a2e95241d94f89a in term 2.
I20240503 02:12:44.255831  5505 raft_consensus.cc:504] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:44.256484  7093 leader_election.cc:302] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 207a8023eda24dc89a2e95241d94f89a, 37c60eb66a6b45aaac4e0d7061572da6; no voters: 
I20240503 02:12:44.256784  5505 leader_election.cc:288] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [CANDIDATE]: Term 2 election: Requested vote from peers 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599), 207a8023eda24dc89a2e95241d94f89a (127.0.145.8:34161)
I20240503 02:12:44.257272  7399 raft_consensus.cc:2781] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:44.257452  4322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27609e0e32a34efc915bbff6750f8234" candidate_uuid: "37c60eb66a6b45aaac4e0d7061572da6" candidate_term: 2 candidate_status { last_received { term: 1 index: 387 } } ignore_live_leader: true dest_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc"
I20240503 02:12:44.257725  4322 raft_consensus.cc:3032] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:44.257539  7158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27609e0e32a34efc915bbff6750f8234" candidate_uuid: "37c60eb66a6b45aaac4e0d7061572da6" candidate_term: 2 candidate_status { last_received { term: 1 index: 387 } } ignore_live_leader: true dest_uuid: "207a8023eda24dc89a2e95241d94f89a"
I20240503 02:12:44.257772  7399 raft_consensus.cc:686] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [term 2 LEADER]: Becoming Leader. State: Replica: 207a8023eda24dc89a2e95241d94f89a, State: Running, Role: LEADER
I20240503 02:12:44.257827  4322 raft_consensus.cc:729] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b812251ceb44fbe8a54fea04a7f8dfc, State: Running, Role: LEADER
I20240503 02:12:44.257961  7158 raft_consensus.cc:3037] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:44.258105  4322 consensus_queue.cc:260] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 387, Committed index: 387, Last appended: 1.387, Last appended by leader: 387, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:44.258515  4322 raft_consensus.cc:3037] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:44.258335  7399 consensus_queue.cc:237] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 392, Committed index: 392, Last appended: 1.392, Last appended by leader: 392, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:44.260654  7158 raft_consensus.cc:2454] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37c60eb66a6b45aaac4e0d7061572da6 in term 2.
I20240503 02:12:44.261132  4322 raft_consensus.cc:2454] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37c60eb66a6b45aaac4e0d7061572da6 in term 2.
I20240503 02:12:44.261204  5439 leader_election.cc:302] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 207a8023eda24dc89a2e95241d94f89a, 37c60eb66a6b45aaac4e0d7061572da6; no voters: 
I20240503 02:12:44.261705  7544 raft_consensus.cc:2781] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:44.262140  7544 raft_consensus.cc:686] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 LEADER]: Becoming Leader. State: Replica: 37c60eb66a6b45aaac4e0d7061572da6, State: Running, Role: LEADER
I20240503 02:12:44.262521  7544 consensus_queue.cc:237] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 387, Committed index: 387, Last appended: 1.387, Last appended by leader: 387, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:44.264043  3919 catalog_manager.cc:5484] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a reported cstate change: term changed from 1 to 2, leader changed from c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) to 207a8023eda24dc89a2e95241d94f89a (127.0.145.8). New cstate: current_term: 2 leader_uuid: "207a8023eda24dc89a2e95241d94f89a" committed_config { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:44.265719  3921 catalog_manager.cc:5484] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 reported cstate change: term changed from 1 to 2, leader changed from 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) to 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5). New cstate: current_term: 2 leader_uuid: "37c60eb66a6b45aaac4e0d7061572da6" committed_config { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:44.643746  4062 raft_consensus.cc:1261] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Refusing update from remote peer 207a8023eda24dc89a2e95241d94f89a: Log matching property violated. Preceding OpId in replica: term: 1 index: 392. Preceding OpId from leader: term: 2 index: 393. (index mismatch)
I20240503 02:12:44.644578  7399 consensus_queue.cc:1035] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 393, Last known committed idx: 392, Time since last communication: 0.000s
I20240503 02:12:44.649909  5504 raft_consensus.cc:1261] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 FOLLOWER]: Refusing update from remote peer 207a8023eda24dc89a2e95241d94f89a: Log matching property violated. Preceding OpId in replica: term: 1 index: 392. Preceding OpId from leader: term: 2 index: 393. (index mismatch)
I20240503 02:12:44.652120  7589 consensus_queue.cc:1035] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 393, Last known committed idx: 392, Time since last communication: 0.000s
I20240503 02:12:44.694684  4322 raft_consensus.cc:1261] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 2 FOLLOWER]: Refusing update from remote peer 37c60eb66a6b45aaac4e0d7061572da6: Log matching property violated. Preceding OpId in replica: term: 1 index: 387. Preceding OpId from leader: term: 2 index: 388. (index mismatch)
I20240503 02:12:44.695374  7544 consensus_queue.cc:1035] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b812251ceb44fbe8a54fea04a7f8dfc" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41599 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20240503 02:12:44.699090  7158 raft_consensus.cc:1261] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a [term 2 FOLLOWER]: Refusing update from remote peer 37c60eb66a6b45aaac4e0d7061572da6: Log matching property violated. Preceding OpId in replica: term: 1 index: 387. Preceding OpId from leader: term: 2 index: 388. (index mismatch)
I20240503 02:12:44.699913  7617 consensus_queue.cc:1035] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20240503 02:12:45.123862  7158 consensus_queue.cc:237] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 393, Committed index: 393, Last appended: 2.393, Last appended by leader: 392, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:45.125738  5504 raft_consensus.cc:1261] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 FOLLOWER]: Refusing update from remote peer 207a8023eda24dc89a2e95241d94f89a: Log matching property violated. Preceding OpId in replica: term: 2 index: 393. Preceding OpId from leader: term: 2 index: 394. (index mismatch)
I20240503 02:12:45.126305  7640 consensus_queue.cc:1035] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 394, Last known committed idx: 393, Time since last communication: 0.000s
I20240503 02:12:45.128727  7399 raft_consensus.cc:2932] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a [term 2 LEADER]: Committing config change with OpId 2.394: config changed from index 392 to 394, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New config: { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:45.129400  5504 raft_consensus.cc:2932] T 8b76c95bac0f497f8ab21c5295028026 P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 FOLLOWER]: Committing config change with OpId 2.394: config changed from index 392 to 394, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New config: { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:45.132826  3907 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8b76c95bac0f497f8ab21c5295028026 with cas_config_opid_index 392: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:45.134105  3921 catalog_manager.cc:5484] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a reported cstate change: config changed from index 392 to 394, VOTER c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1) evicted. New cstate: current_term: 2 leader_uuid: "207a8023eda24dc89a2e95241d94f89a" committed_config { opid_index: 394 OBSOLETE_local: false peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:45.139225  4043 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b76c95bac0f497f8ab21c5295028026 with delete_type TABLET_DATA_TOMBSTONED (TS c4af56cee8e546c384cf20f40deeacd7 not found in new config with opid_index 394) from {username='slave'} at 127.0.0.1:51996
I20240503 02:12:45.139954  7645 tablet_replica.cc:330] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7: stopping tablet replica
I20240503 02:12:45.140167  7645 raft_consensus.cc:2229] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:45.140360  7645 raft_consensus.cc:2258] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:45.142226  7645 ts_tablet_manager.cc:1875] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:45.146857  7645 ts_tablet_manager.cc:1888] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.393
I20240503 02:12:45.146984  7645 log.cc:1196] T 8b76c95bac0f497f8ab21c5295028026 P c4af56cee8e546c384cf20f40deeacd7: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/8b76c95bac0f497f8ab21c5295028026
I20240503 02:12:45.147833  3909 catalog_manager.cc:4830] TS c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835): tablet 8b76c95bac0f497f8ab21c5295028026 (table rebalance_test_table_3 [id=76e610e6ec844c2da65036194e355bbe]) successfully deleted
I20240503 02:12:45.239454  5504 consensus_queue.cc:237] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 388, Committed index: 388, Last appended: 2.388, Last appended by leader: 387, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:45.241088  7158 raft_consensus.cc:1261] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a [term 2 FOLLOWER]: Refusing update from remote peer 37c60eb66a6b45aaac4e0d7061572da6: Log matching property violated. Preceding OpId in replica: term: 2 index: 388. Preceding OpId from leader: term: 2 index: 389. (index mismatch)
I20240503 02:12:45.241600  7648 consensus_queue.cc:1035] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I20240503 02:12:45.244590  7647 raft_consensus.cc:2932] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 [term 2 LEADER]: Committing config change with OpId 2.389: config changed from index 387 to 389, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:45.245016  7158 raft_consensus.cc:2932] T 27609e0e32a34efc915bbff6750f8234 P 207a8023eda24dc89a2e95241d94f89a [term 2 FOLLOWER]: Committing config change with OpId 2.389: config changed from index 387 to 389, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } } }
I20240503 02:12:45.249361  3908 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 27609e0e32a34efc915bbff6750f8234 with cas_config_opid_index 387: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:45.250690  3921 catalog_manager.cc:5484] T 27609e0e32a34efc915bbff6750f8234 P 37c60eb66a6b45aaac4e0d7061572da6 reported cstate change: config changed from index 387 to 389, VOTER 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3) evicted. New cstate: current_term: 2 leader_uuid: "37c60eb66a6b45aaac4e0d7061572da6" committed_config { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "37c60eb66a6b45aaac4e0d7061572da6" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 39125 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:45.254756  4305 tablet_service.cc:1514] Processing DeleteTablet for tablet 27609e0e32a34efc915bbff6750f8234 with delete_type TABLET_DATA_TOMBSTONED (TS 8b812251ceb44fbe8a54fea04a7f8dfc not found in new config with opid_index 389) from {username='slave'} at 127.0.0.1:54992
I20240503 02:12:45.255402  7655 tablet_replica.cc:330] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc: stopping tablet replica
I20240503 02:12:45.255612  7655 raft_consensus.cc:2229] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:45.255812  7655 raft_consensus.cc:2258] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:45.257781  7655 ts_tablet_manager.cc:1875] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:45.263293  7655 ts_tablet_manager.cc:1888] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.388
I20240503 02:12:45.263447  7655 log.cc:1196] T 27609e0e32a34efc915bbff6750f8234 P 8b812251ceb44fbe8a54fea04a7f8dfc: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/27609e0e32a34efc915bbff6750f8234
I20240503 02:12:45.264235  3907 catalog_manager.cc:4830] TS 8b812251ceb44fbe8a54fea04a7f8dfc (127.0.145.3:41599): tablet 27609e0e32a34efc915bbff6750f8234 (table rebalance_test_table_4 [id=7bd39e7281604b6ebd6334a68999befc]) successfully deleted
I20240503 02:12:45.382409  7138 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:45.409756  5745 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:45.416126  5485 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:45.436902  5354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:45.443338  4305 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:45.444870  5615 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:45.466745  4043 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:45.503248  4173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:45.672433  7138 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:45.673360  5485 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:45.699241  5745 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:45.711268  5354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:45.714156  4305 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:45.742127  5615 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:45.744066  4043 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:45.763561  4173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:46.106633  7138 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:46.122319  4305 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:46.128703  5615 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:46.137486  5485 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:46.143261  4173 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:46.172931  5354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:46.173101  5745 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:46.178936  4043 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 1a0bcd52c0054aa6a0012cbd524d103c | 127.0.145.62:40001 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                               Value                                                                                |         Tags          |         Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                               | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                               | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                  | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                               | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                               | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                               | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                 | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.0.145.20:36429 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address      | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
 207a8023eda24dc89a2e95241d94f89a | 127.0.145.8:34161 | HEALTHY | <none>   |       1        |       0
 37c60eb66a6b45aaac4e0d7061572da6 | 127.0.145.5:39125 | HEALTHY | <none>   |       2        |       0
 6c3559ccff5d4c6681b3c90f892143dc | 127.0.145.7:35457 | HEALTHY | <none>   |       0        |       0
 6ebeaff495e042c78067c5b18600dc98 | 127.0.145.4:33659 | HEALTHY | <none>   |       0        |       0
 8b812251ceb44fbe8a54fea04a7f8dfc | 127.0.145.3:41599 | HEALTHY | <none>   |       4        |       0
 9c804fe67ee34e7fae5fab23db90f298 | 127.0.145.6:36423 | HEALTHY | <none>   |       1        |       0
 c4af56cee8e546c384cf20f40deeacd7 | 127.0.145.1:43835 | HEALTHY | <none>   |       4        |       0
 d7f19d54f56849f495dc547de6a47ad2 | 127.0.145.2:44433 | HEALTHY | <none>   |       4        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       8

Unusual flags for Tablet Server:
                   Flag                   |                                                                             Value                                                                              |         Tags          |      Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                            | experimental          | all 8 server(s) checked
 local_ip_for_outbound_sockets            | 127.0.145.1                                                                                                                                                    | experimental          | 127.0.145.1:43835
 local_ip_for_outbound_sockets            | 127.0.145.2                                                                                                                                                    | experimental          | 127.0.145.2:44433
 local_ip_for_outbound_sockets            | 127.0.145.3                                                                                                                                                    | experimental          | 127.0.145.3:41599
 local_ip_for_outbound_sockets            | 127.0.145.4                                                                                                                                                    | experimental          | 127.0.145.4:33659
 local_ip_for_outbound_sockets            | 127.0.145.5                                                                                                                                                    | experimental          | 127.0.145.5:39125
 local_ip_for_outbound_sockets            | 127.0.145.6                                                                                                                                                    | experimental          | 127.0.145.6:36423
 local_ip_for_outbound_sockets            | 127.0.145.7                                                                                                                                                    | experimental          | 127.0.145.7:35457
 local_ip_for_outbound_sockets            | 127.0.145.8                                                                                                                                                    | experimental          | 127.0.145.8:34161
 never_fsync                              | true                                                                                                                                                           | unsafe,advanced       | all 8 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                              | unsafe,hidden         | all 8 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                           | experimental,advanced | all 8 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                           | unsafe                | all 8 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                             | hidden                | all 8 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.0.145.1:43835
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.0.145.2:44433
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.0.145.3:41599
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.0.145.4:33659
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.0.145.5:39125
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.0.145.6:36423
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.0.145.7:35457
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden                | 127.0.145.8:34161

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.0.145.20:36429 | all 8 server(s) checked
 time_source         | builtin            | all 8 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 9 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 4
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 8
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240503 02:12:46.255518   580 log_verifier.cc:126] Checking tablet 07df29aeb41b427a8dcaf4a1a7db30da
I20240503 02:12:46.417447   580 log_verifier.cc:177] Verified matching terms for 421 ops in tablet 07df29aeb41b427a8dcaf4a1a7db30da
I20240503 02:12:46.417989   580 log_verifier.cc:126] Checking tablet 15fafdc84fec4b46979c311f377fd312
I20240503 02:12:46.569300   580 log_verifier.cc:177] Verified matching terms for 411 ops in tablet 15fafdc84fec4b46979c311f377fd312
I20240503 02:12:46.569821   580 log_verifier.cc:126] Checking tablet 1c1ff96ff85b4d5e9bd80fe5838b16db
I20240503 02:12:46.585726   580 log_verifier.cc:177] Verified matching terms for 8 ops in tablet 1c1ff96ff85b4d5e9bd80fe5838b16db
I20240503 02:12:46.586030   580 log_verifier.cc:126] Checking tablet 27609e0e32a34efc915bbff6750f8234
I20240503 02:12:46.741667   580 log_verifier.cc:177] Verified matching terms for 389 ops in tablet 27609e0e32a34efc915bbff6750f8234
I20240503 02:12:46.742164   580 log_verifier.cc:126] Checking tablet 368fca99d20d4d6e826c45e44379c056
I20240503 02:12:46.872689   580 log_verifier.cc:177] Verified matching terms for 323 ops in tablet 368fca99d20d4d6e826c45e44379c056
I20240503 02:12:46.873221   580 log_verifier.cc:126] Checking tablet 6e9898815c904092af0a6f238ed961bd
I20240503 02:12:47.008736   580 log_verifier.cc:177] Verified matching terms for 379 ops in tablet 6e9898815c904092af0a6f238ed961bd
I20240503 02:12:47.009202   580 log_verifier.cc:126] Checking tablet 75a2da7ca68b43e28cad0847e4d27982
I20240503 02:12:47.157903   580 log_verifier.cc:177] Verified matching terms for 428 ops in tablet 75a2da7ca68b43e28cad0847e4d27982
I20240503 02:12:47.158427   580 log_verifier.cc:126] Checking tablet 8334f743021f4957a427800af656bec1
I20240503 02:12:47.310729   580 log_verifier.cc:177] Verified matching terms for 416 ops in tablet 8334f743021f4957a427800af656bec1
I20240503 02:12:47.311242   580 log_verifier.cc:126] Checking tablet 87e472f1975b4141b795aa6c3b75c7c0
I20240503 02:12:47.473037   580 log_verifier.cc:177] Verified matching terms for 429 ops in tablet 87e472f1975b4141b795aa6c3b75c7c0
I20240503 02:12:47.473536   580 log_verifier.cc:126] Checking tablet 8b76c95bac0f497f8ab21c5295028026
I20240503 02:12:47.613103   580 log_verifier.cc:177] Verified matching terms for 394 ops in tablet 8b76c95bac0f497f8ab21c5295028026
I20240503 02:12:47.613620   580 log_verifier.cc:126] Checking tablet ccccc26d94514b87a91b0c00fa18e669
I20240503 02:12:47.750278   580 log_verifier.cc:177] Verified matching terms for 390 ops in tablet ccccc26d94514b87a91b0c00fa18e669
I20240503 02:12:47.750831   580 log_verifier.cc:126] Checking tablet d846ad6206514c79b142064043ad1cb0
I20240503 02:12:47.876402   580 log_verifier.cc:177] Verified matching terms for 336 ops in tablet d846ad6206514c79b142064043ad1cb0
I20240503 02:12:47.876864   580 log_verifier.cc:126] Checking tablet d98afd14a7f14415a0f404dea8750d82
I20240503 02:12:48.033922   580 log_verifier.cc:177] Verified matching terms for 393 ops in tablet d98afd14a7f14415a0f404dea8750d82
I20240503 02:12:48.034438   580 log_verifier.cc:126] Checking tablet df156172a4554094b36cd5cc3876c922
I20240503 02:12:48.165724   580 log_verifier.cc:177] Verified matching terms for 381 ops in tablet df156172a4554094b36cd5cc3876c922
I20240503 02:12:48.166227   580 log_verifier.cc:126] Checking tablet ec5c9e506c2f4c29aea9a9de259a0c19
I20240503 02:12:48.290747   580 log_verifier.cc:177] Verified matching terms for 332 ops in tablet ec5c9e506c2f4c29aea9a9de259a0c19
I20240503 02:12:48.291221   580 log_verifier.cc:126] Checking tablet ed79dde0a1f4419c8085c042ed5b8780
I20240503 02:12:48.420671   580 log_verifier.cc:177] Verified matching terms for 352 ops in tablet ed79dde0a1f4419c8085c042ed5b8780
I20240503 02:12:48.429549   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 3981
I20240503 02:12:48.430770  4103 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:48.674928   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 3981
W20240503 02:12:48.720883  3906 connection.cc:535] server connection from 127.0.145.1:45305 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:48.721084   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 4112
I20240503 02:12:48.722102  4233 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:48.947284   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 4112
W20240503 02:12:48.987788  5701 connection.cc:535] server connection from 127.0.145.2:39019 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:48.988632   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 4243
I20240503 02:12:48.989595  4365 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:49.213351   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 4243
W20240503 02:12:49.255026  5571 connection.cc:535] server connection from 127.0.145.3:56301 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:49.255662   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 5292
I20240503 02:12:49.272866  5413 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:49.434391   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 5292
W20240503 02:12:49.465695  3908 connection.cc:535] server connection from 127.0.145.4:55775 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:49.465996   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 5423
I20240503 02:12:49.472342  5544 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:49.633114   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 5423
I20240503 02:12:49.637110  7767 raft_consensus.cc:482] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c4af56cee8e546c384cf20f40deeacd7)
I20240503 02:12:49.637333  7767 raft_consensus.cc:504] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "207a8023eda24dc89a2e95241d94f89a" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 34161 } attrs { promote: false } }
I20240503 02:12:49.637969  7767 leader_election.cc:288] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
W20240503 02:12:49.639128  7094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.1:43835: connect: Connection refused (error 111)
W20240503 02:12:49.643234  7094 leader_election.cc:334] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835): Network error: Client connection negotiation failed: client connection to 127.0.145.1:43835: connect: Connection refused (error 111)
I20240503 02:12:49.643484  7094 leader_election.cc:302] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 207a8023eda24dc89a2e95241d94f89a; no voters: c4af56cee8e546c384cf20f40deeacd7
I20240503 02:12:49.643839  7767 raft_consensus.cc:2726] T d846ad6206514c79b142064043ad1cb0 P 207a8023eda24dc89a2e95241d94f89a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:49.665565   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 5553
I20240503 02:12:49.671797  5675 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:49.773347  7783 raft_consensus.cc:482] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c4af56cee8e546c384cf20f40deeacd7)
I20240503 02:12:49.773572  7783 raft_consensus.cc:504] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:49.775048  7783 leader_election.cc:288] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
W20240503 02:12:49.779448  5701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.1:43835: connect: Connection refused (error 111)
W20240503 02:12:49.781018  5701 leader_election.cc:334] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835): Network error: Client connection negotiation failed: client connection to 127.0.145.1:43835: connect: Connection refused (error 111)
I20240503 02:12:49.781176  5701 leader_election.cc:302] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6c3559ccff5d4c6681b3c90f892143dc; no voters: c4af56cee8e546c384cf20f40deeacd7
I20240503 02:12:49.781509  7783 raft_consensus.cc:2726] T 75a2da7ca68b43e28cad0847e4d27982 P 6c3559ccff5d4c6681b3c90f892143dc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:12:49.845017  7786 raft_consensus.cc:482] T ccccc26d94514b87a91b0c00fa18e669 P 9c804fe67ee34e7fae5fab23db90f298 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c4af56cee8e546c384cf20f40deeacd7)
I20240503 02:12:49.845145   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 5553
I20240503 02:12:49.869493  7783 raft_consensus.cc:482] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c4af56cee8e546c384cf20f40deeacd7)
I20240503 02:12:49.869735  7783 raft_consensus.cc:504] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "c4af56cee8e546c384cf20f40deeacd7" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 43835 } } peers { permanent_uuid: "6c3559ccff5d4c6681b3c90f892143dc" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 35457 } attrs { promote: false } }
I20240503 02:12:49.870276  7783 leader_election.cc:288] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835)
W20240503 02:12:49.871778  5701 leader_election.cc:334] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c4af56cee8e546c384cf20f40deeacd7 (127.0.145.1:43835): Network error: Client connection negotiation failed: client connection to 127.0.145.1:43835: connect: Connection refused (error 111)
I20240503 02:12:49.871933  5701 leader_election.cc:302] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6c3559ccff5d4c6681b3c90f892143dc; no voters: c4af56cee8e546c384cf20f40deeacd7
I20240503 02:12:49.872217  7783 raft_consensus.cc:2726] T 15fafdc84fec4b46979c311f377fd312 P 6c3559ccff5d4c6681b3c90f892143dc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:12:49.878198   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 5683
I20240503 02:12:49.884243  5804 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:12:49.903509  7093 consensus_peers.cc:485] T 8b76c95bac0f497f8ab21c5295028026 P 207a8023eda24dc89a2e95241d94f89a -> Peer 37c60eb66a6b45aaac4e0d7061572da6 (127.0.145.5:39125): Couldn't send request to peer 37c60eb66a6b45aaac4e0d7061572da6. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:39125: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:50.048465   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 5683
I20240503 02:12:50.080014   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 7075
I20240503 02:12:50.112356  7198 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:50.233879   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 7075
I20240503 02:12:50.255178   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 3891
I20240503 02:12:50.256372  3952 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:50.421519   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 3891
2024-05-03T02:12:50Z chronyd exiting
[       OK ] RebalanceParamTest.Rebalance/3 (54694 ms)
[----------] 1 test from RebalanceParamTest (54694 ms total)

[----------] 1 test from ConcurrentRebalancersTest
[ RUN      ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240503 02:12:50.513639   580 test_util.cc:268] Using random seed: 1102095066
I20240503 02:12:50.515465   580 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:12:50.515525   580 ts_itest-base.cc:113] --------------
I20240503 02:12:50.515547   580 ts_itest-base.cc:114] 8 tablet servers
I20240503 02:12:50.515583   580 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:12:50.515610   580 ts_itest-base.cc:116] --------------
2024-05-03T02:12:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:12:50Z Disabled control of system clock
I20240503 02:12:50.565625   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.145.62:36865
--webserver_interface=127.0.145.62
--webserver_port=0
--builtin_ntp_servers=127.0.145.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.145.62:36865
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 02:12:50.919241  7799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:50.919595  7799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:50.919732  7799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:50.927472  7799 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:50.927623  7799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:12:50.927711  7799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:50.927788  7799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:12:50.927861  7799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:12:50.937330  7799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.145.62:36865
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.145.62:36865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.145.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:50.938766  7799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:50.940637  7799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:50.952566  7807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:50.953388  7804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:50.954202  7805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:50.955221  7799 server_base.cc:1034] running on GCE node
I20240503 02:12:50.956110  7799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:50.957659  7799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:50.958884  7799 hybrid_clock.cc:648] HybridClock initialized: now 1714702370958837 us; error 41 us; skew 500 ppm
I20240503 02:12:50.959247  7799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:50.961647  7799 webserver.cc:460] Webserver started at http://127.0.145.62:33115/ using document root <none> and password file <none>
I20240503 02:12:50.962428  7799 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:50.962553  7799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:50.962901  7799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:50.965695  7799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5680e1b3edfe402b831ab3889e06b093"
format_stamp: "Formatted at 2024-05-03 02:12:50 on dist-test-slave-gq9h"
I20240503 02:12:50.966506  7799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5680e1b3edfe402b831ab3889e06b093"
format_stamp: "Formatted at 2024-05-03 02:12:50 on dist-test-slave-gq9h"
I20240503 02:12:50.971989  7799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:12:50.975548  7813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:50.976992  7799 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 02:12:50.977190  7799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5680e1b3edfe402b831ab3889e06b093"
format_stamp: "Formatted at 2024-05-03 02:12:50 on dist-test-slave-gq9h"
I20240503 02:12:50.977411  7799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:51.002303  7799 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:51.003306  7799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:51.003609  7799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:51.022521  7799 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.62:36865
I20240503 02:12:51.022544  7864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.62:36865 every 8 connection(s)
I20240503 02:12:51.023993  7799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:12:51.026850   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 7799
I20240503 02:12:51.027037   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:12:51.027858  7865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.040402  7865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093: Bootstrap starting.
I20240503 02:12:51.043716  7865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.044835  7865 log.cc:824] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:51.047333  7865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093: No bootstrap required, opened a new log
I20240503 02:12:51.053653  7865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5680e1b3edfe402b831ab3889e06b093" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 36865 } }
I20240503 02:12:51.054056  7865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.054209  7865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5680e1b3edfe402b831ab3889e06b093, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.054914  7865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5680e1b3edfe402b831ab3889e06b093" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 36865 } }
I20240503 02:12:51.055136  7865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:12:51.055279  7865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:12:51.055454  7865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.057570  7865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5680e1b3edfe402b831ab3889e06b093" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 36865 } }
I20240503 02:12:51.058079  7865 leader_election.cc:302] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5680e1b3edfe402b831ab3889e06b093; no voters: 
I20240503 02:12:51.058737  7865 leader_election.cc:288] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:12:51.058888  7870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:51.059536  7870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [term 1 LEADER]: Becoming Leader. State: Replica: 5680e1b3edfe402b831ab3889e06b093, State: Running, Role: LEADER
I20240503 02:12:51.060096  7870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5680e1b3edfe402b831ab3889e06b093" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 36865 } }
I20240503 02:12:51.060801  7865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:51.063004  7872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5680e1b3edfe402b831ab3889e06b093. Latest consensus state: current_term: 1 leader_uuid: "5680e1b3edfe402b831ab3889e06b093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5680e1b3edfe402b831ab3889e06b093" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 36865 } } }
I20240503 02:12:51.063445  7872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:51.063838  7871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5680e1b3edfe402b831ab3889e06b093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5680e1b3edfe402b831ab3889e06b093" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 36865 } } }
I20240503 02:12:51.064095  7871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093 [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:51.066005  7877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:12:51.071755  7877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:12:51.080646  7877 catalog_manager.cc:1347] Generated new cluster ID: 6a8c4e6031984d50aee85294eb2c175d
I20240503 02:12:51.080780  7877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:12:51.094784  7877 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:12:51.096639  7877 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:12:51.121277  7877 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5680e1b3edfe402b831ab3889e06b093: Generated new TSK 0
I20240503 02:12:51.122242  7877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:12:51.137243   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.1:0
--local_ip_for_outbound_sockets=127.0.145.1
--webserver_interface=127.0.145.1
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--builtin_ntp_servers=127.0.145.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:12:51.492810  7889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:51.493135  7889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:51.493237  7889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:51.500630  7889 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:51.500764  7889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:51.500931  7889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.1
I20240503 02:12:51.510047  7889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.145.1
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.1
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:51.511480  7889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:51.513288  7889 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:51.528683  7894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:51.529556  7895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:51.530508  7897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:51.531459  7889 server_base.cc:1034] running on GCE node
I20240503 02:12:51.532140  7889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:51.533152  7889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:51.534350  7889 hybrid_clock.cc:648] HybridClock initialized: now 1714702371534288 us; error 59 us; skew 500 ppm
I20240503 02:12:51.534737  7889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:51.537112  7889 webserver.cc:460] Webserver started at http://127.0.145.1:39851/ using document root <none> and password file <none>
I20240503 02:12:51.537842  7889 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:51.537947  7889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:51.538236  7889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:51.540963  7889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d986fcfb926e47baa7d2d5b99f697334"
format_stamp: "Formatted at 2024-05-03 02:12:51 on dist-test-slave-gq9h"
I20240503 02:12:51.541805  7889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d986fcfb926e47baa7d2d5b99f697334"
format_stamp: "Formatted at 2024-05-03 02:12:51 on dist-test-slave-gq9h"
I20240503 02:12:51.547435  7889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
I20240503 02:12:51.550999  7903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:51.552575  7889 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 02:12:51.552733  7889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d986fcfb926e47baa7d2d5b99f697334"
format_stamp: "Formatted at 2024-05-03 02:12:51 on dist-test-slave-gq9h"
I20240503 02:12:51.552920  7889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:51.573160  7889 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:51.574028  7889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:51.574272  7889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:51.575273  7889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:51.576740  7889 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:51.576821  7889 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:51.576902  7889 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:51.576956  7889 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:51.607868  7889 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.1:37681
I20240503 02:12:51.607923  8015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.1:37681 every 8 connection(s)
I20240503 02:12:51.609275  7889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:12:51.612836   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 7889
I20240503 02:12:51.613088   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:12:51.616941   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.2:0
--local_ip_for_outbound_sockets=127.0.145.2
--webserver_interface=127.0.145.2
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--builtin_ntp_servers=127.0.145.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:51.623378  8016 heartbeater.cc:347] Connected to a master server at 127.0.145.62:36865
I20240503 02:12:51.623718  8016 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:51.624632  8016 heartbeater.cc:511] Master 127.0.145.62:36865 requested a full tablet report, sending...
I20240503 02:12:51.626808  7830 ts_manager.cc:194] Registered new tserver with Master: d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:12:51.628311  7830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.1:43665
W20240503 02:12:51.968351  8020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:51.968698  8020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:51.968891  8020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:51.977931  8020 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:51.978086  8020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:51.978257  8020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.2
I20240503 02:12:51.987676  8020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.145.2
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.2
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:51.989084  8020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:51.990978  8020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:52.004280  8026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:52.004784  8025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:52.005787  8028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:52.006610  8020 server_base.cc:1034] running on GCE node
I20240503 02:12:52.007305  8020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:52.008394  8020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:52.009606  8020 hybrid_clock.cc:648] HybridClock initialized: now 1714702372009548 us; error 51 us; skew 500 ppm
I20240503 02:12:52.009958  8020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:52.012387  8020 webserver.cc:460] Webserver started at http://127.0.145.2:46393/ using document root <none> and password file <none>
I20240503 02:12:52.013175  8020 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:52.013288  8020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:52.013653  8020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:52.016520  8020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
format_stamp: "Formatted at 2024-05-03 02:12:52 on dist-test-slave-gq9h"
I20240503 02:12:52.017324  8020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
format_stamp: "Formatted at 2024-05-03 02:12:52 on dist-test-slave-gq9h"
I20240503 02:12:52.023241  8020 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.003s	sys 0.005s
I20240503 02:12:52.027817  8034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:52.029603  8020 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:12:52.029817  8020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
format_stamp: "Formatted at 2024-05-03 02:12:52 on dist-test-slave-gq9h"
I20240503 02:12:52.030066  8020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:52.070690  8020 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:52.071576  8020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:52.071841  8020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:52.072861  8020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:52.074398  8020 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:52.074509  8020 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:52.074620  8020 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:52.074704  8020 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20240503 02:12:52.107885  8020 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.2:35309
I20240503 02:12:52.107951  8146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.2:35309 every 8 connection(s)
I20240503 02:12:52.109292  8020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:12:52.117817   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 8020
I20240503 02:12:52.117983   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:12:52.121454   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.3:0
--local_ip_for_outbound_sockets=127.0.145.3
--webserver_interface=127.0.145.3
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--builtin_ntp_servers=127.0.145.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:52.123574  8147 heartbeater.cc:347] Connected to a master server at 127.0.145.62:36865
I20240503 02:12:52.123911  8147 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:52.124815  8147 heartbeater.cc:511] Master 127.0.145.62:36865 requested a full tablet report, sending...
I20240503 02:12:52.126502  7830 ts_manager.cc:194] Registered new tserver with Master: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:12:52.127656  7830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.2:58865
W20240503 02:12:52.471468  8150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:52.471773  8150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:52.471920  8150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:52.479533  8150 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:52.479681  8150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:52.479851  8150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.3
I20240503 02:12:52.489655  8150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.145.3
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.3
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:52.491371  8150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:52.493275  8150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:52.506263  8155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:52.506908  8156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:52.507894  8158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:52.508781  8150 server_base.cc:1034] running on GCE node
I20240503 02:12:52.509444  8150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:52.510475  8150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:52.511682  8150 hybrid_clock.cc:648] HybridClock initialized: now 1714702372511633 us; error 44 us; skew 500 ppm
I20240503 02:12:52.512007  8150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:52.514353  8150 webserver.cc:460] Webserver started at http://127.0.145.3:37693/ using document root <none> and password file <none>
I20240503 02:12:52.515208  8150 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:52.515336  8150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:52.515664  8150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:52.518325  8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
format_stamp: "Formatted at 2024-05-03 02:12:52 on dist-test-slave-gq9h"
I20240503 02:12:52.519133  8150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
format_stamp: "Formatted at 2024-05-03 02:12:52 on dist-test-slave-gq9h"
I20240503 02:12:52.524452  8150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:12:52.527930  8164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:52.529335  8150 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:12:52.529512  8150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
format_stamp: "Formatted at 2024-05-03 02:12:52 on dist-test-slave-gq9h"
I20240503 02:12:52.529712  8150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:52.543828  8150 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:52.544687  8150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:52.544970  8150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:52.545919  8150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:52.547489  8150 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:52.547581  8150 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:52.547699  8150 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:52.547780  8150 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:52.580093  8150 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.3:38203
I20240503 02:12:52.580147  8276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.3:38203 every 8 connection(s)
I20240503 02:12:52.581482  8150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:12:52.583329   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 8150
I20240503 02:12:52.583527   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:12:52.586500   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.4:0
--local_ip_for_outbound_sockets=127.0.145.4
--webserver_interface=127.0.145.4
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--builtin_ntp_servers=127.0.145.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:52.594830  8277 heartbeater.cc:347] Connected to a master server at 127.0.145.62:36865
I20240503 02:12:52.595163  8277 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:52.596056  8277 heartbeater.cc:511] Master 127.0.145.62:36865 requested a full tablet report, sending...
I20240503 02:12:52.597599  7830 ts_manager.cc:194] Registered new tserver with Master: 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:12:52.598815  7830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.3:47937
I20240503 02:12:52.631363  8016 heartbeater.cc:503] Master 127.0.145.62:36865 was elected leader, sending a full tablet report...
W20240503 02:12:52.938066  8280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:52.938402  8280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:52.938539  8280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:52.946449  8280 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:52.946645  8280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:52.946832  8280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.4
I20240503 02:12:52.956558  8280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.145.4
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.4
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:52.958079  8280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:52.960013  8280 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:52.973644  8285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:52.974311  8286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:52.975327  8288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:52.976166  8280 server_base.cc:1034] running on GCE node
I20240503 02:12:52.976850  8280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:52.977895  8280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:52.979104  8280 hybrid_clock.cc:648] HybridClock initialized: now 1714702372979052 us; error 48 us; skew 500 ppm
I20240503 02:12:52.979442  8280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:52.981894  8280 webserver.cc:460] Webserver started at http://127.0.145.4:42269/ using document root <none> and password file <none>
I20240503 02:12:52.982743  8280 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:52.982872  8280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:52.983202  8280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:52.985935  8280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "8d7c037d50c241b484ee843b06313a64"
format_stamp: "Formatted at 2024-05-03 02:12:52 on dist-test-slave-gq9h"
I20240503 02:12:52.986817  8280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "8d7c037d50c241b484ee843b06313a64"
format_stamp: "Formatted at 2024-05-03 02:12:52 on dist-test-slave-gq9h"
I20240503 02:12:52.992362  8280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.000s	sys 0.004s
I20240503 02:12:52.996277  8294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:52.997784  8280 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.001s
I20240503 02:12:52.998001  8280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8d7c037d50c241b484ee843b06313a64"
format_stamp: "Formatted at 2024-05-03 02:12:52 on dist-test-slave-gq9h"
I20240503 02:12:52.998222  8280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:53.011828  8280 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:53.012758  8280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:53.013053  8280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:53.014165  8280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:53.015889  8280 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:53.015983  8280 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:53.016078  8280 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:53.016134  8280 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:53.050804  8280 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.4:35377
I20240503 02:12:53.050866  8406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.4:35377 every 8 connection(s)
I20240503 02:12:53.052266  8280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:12:53.058578   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 8280
I20240503 02:12:53.058821   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:12:53.062386   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.5:0
--local_ip_for_outbound_sockets=127.0.145.5
--webserver_interface=127.0.145.5
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--builtin_ntp_servers=127.0.145.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:53.066541  8407 heartbeater.cc:347] Connected to a master server at 127.0.145.62:36865
I20240503 02:12:53.066922  8407 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:53.068078  8407 heartbeater.cc:511] Master 127.0.145.62:36865 requested a full tablet report, sending...
I20240503 02:12:53.069644  7830 ts_manager.cc:194] Registered new tserver with Master: 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:12:53.070456  7830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.4:49523
I20240503 02:12:53.130421  8147 heartbeater.cc:503] Master 127.0.145.62:36865 was elected leader, sending a full tablet report...
W20240503 02:12:53.420269  8410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:53.420614  8410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:53.420773  8410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:53.428323  8410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:53.428496  8410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:53.428673  8410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.5
I20240503 02:12:53.438329  8410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.145.5
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.5
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:53.439800  8410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:53.441603  8410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:53.454420  8415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:53.455045  8418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:53.456005  8416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:53.456804  8410 server_base.cc:1034] running on GCE node
I20240503 02:12:53.457487  8410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:53.458482  8410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:53.459686  8410 hybrid_clock.cc:648] HybridClock initialized: now 1714702373459628 us; error 49 us; skew 500 ppm
I20240503 02:12:53.460023  8410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:53.462411  8410 webserver.cc:460] Webserver started at http://127.0.145.5:35987/ using document root <none> and password file <none>
I20240503 02:12:53.463202  8410 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:53.463311  8410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:53.463636  8410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:53.466284  8410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "03f99420df704238ab300ad6e58d878c"
format_stamp: "Formatted at 2024-05-03 02:12:53 on dist-test-slave-gq9h"
I20240503 02:12:53.467108  8410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "03f99420df704238ab300ad6e58d878c"
format_stamp: "Formatted at 2024-05-03 02:12:53 on dist-test-slave-gq9h"
I20240503 02:12:53.472270  8410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:12:53.475678  8424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:53.477036  8410 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:12:53.477216  8410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "03f99420df704238ab300ad6e58d878c"
format_stamp: "Formatted at 2024-05-03 02:12:53 on dist-test-slave-gq9h"
I20240503 02:12:53.477411  8410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:53.522240  8410 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:53.523171  8410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:53.523445  8410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:53.524384  8410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:53.525830  8410 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:53.525914  8410 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:53.525998  8410 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:53.526052  8410 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:53.558061  8410 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.5:44891
I20240503 02:12:53.558121  8536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.5:44891 every 8 connection(s)
I20240503 02:12:53.559612  8410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:12:53.564309   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 8410
I20240503 02:12:53.564498   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:12:53.567934   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.6:0
--local_ip_for_outbound_sockets=127.0.145.6
--webserver_interface=127.0.145.6
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--builtin_ntp_servers=127.0.145.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:53.573477  8537 heartbeater.cc:347] Connected to a master server at 127.0.145.62:36865
I20240503 02:12:53.573794  8537 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:53.574721  8537 heartbeater.cc:511] Master 127.0.145.62:36865 requested a full tablet report, sending...
I20240503 02:12:53.576125  7830 ts_manager.cc:194] Registered new tserver with Master: 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891)
I20240503 02:12:53.576857  7830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.5:54473
I20240503 02:12:53.601461  8277 heartbeater.cc:503] Master 127.0.145.62:36865 was elected leader, sending a full tablet report...
W20240503 02:12:53.913475  8540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:53.913769  8540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:53.913910  8540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:53.921566  8540 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:53.921703  8540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:53.921913  8540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.6
I20240503 02:12:53.931253  8540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.145.6
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.6
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:53.932626  8540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:53.934470  8540 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:53.947607  8545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:53.948452  8548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:53.949148  8546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:53.950240  8540 server_base.cc:1034] running on GCE node
I20240503 02:12:53.950917  8540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:53.951876  8540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:53.953087  8540 hybrid_clock.cc:648] HybridClock initialized: now 1714702373953033 us; error 65 us; skew 500 ppm
I20240503 02:12:53.953401  8540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:53.955803  8540 webserver.cc:460] Webserver started at http://127.0.145.6:45689/ using document root <none> and password file <none>
I20240503 02:12:53.956621  8540 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:53.956743  8540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:53.957072  8540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:53.959887  8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "b85e85e631ab4e04a86bfe5a9f0aa512"
format_stamp: "Formatted at 2024-05-03 02:12:53 on dist-test-slave-gq9h"
I20240503 02:12:53.960678  8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "b85e85e631ab4e04a86bfe5a9f0aa512"
format_stamp: "Formatted at 2024-05-03 02:12:53 on dist-test-slave-gq9h"
I20240503 02:12:53.967041  8540 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20240503 02:12:53.971616  8554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:53.973014  8540 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.005s	sys 0.000s
I20240503 02:12:53.973189  8540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "b85e85e631ab4e04a86bfe5a9f0aa512"
format_stamp: "Formatted at 2024-05-03 02:12:53 on dist-test-slave-gq9h"
I20240503 02:12:53.973393  8540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:53.994184  8540 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:53.995158  8540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:53.995447  8540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:53.996403  8540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:53.997933  8540 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:53.998023  8540 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:53.998143  8540 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:53.998219  8540 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:54.031648  8540 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.6:40397
I20240503 02:12:54.031702  8666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.6:40397 every 8 connection(s)
I20240503 02:12:54.032967  8540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:12:54.039407   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 8540
I20240503 02:12:54.039623   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:12:54.043457   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.7:0
--local_ip_for_outbound_sockets=127.0.145.7
--webserver_interface=127.0.145.7
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--builtin_ntp_servers=127.0.145.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:54.047443  8667 heartbeater.cc:347] Connected to a master server at 127.0.145.62:36865
I20240503 02:12:54.047850  8667 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:54.048817  8667 heartbeater.cc:511] Master 127.0.145.62:36865 requested a full tablet report, sending...
I20240503 02:12:54.050666  7830 ts_manager.cc:194] Registered new tserver with Master: b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397)
I20240503 02:12:54.051859  7830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.6:43007
I20240503 02:12:54.073570  8407 heartbeater.cc:503] Master 127.0.145.62:36865 was elected leader, sending a full tablet report...
W20240503 02:12:54.401400  8670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:54.401746  8670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:54.401911  8670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:54.410130  8670 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:54.410290  8670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:54.410463  8670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.7
I20240503 02:12:54.419931  8670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.145.7
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.7
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:54.421314  8670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:54.423190  8670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:54.435899  8676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:54.436514  8675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:54.437474  8678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:54.438359  8670 server_base.cc:1034] running on GCE node
I20240503 02:12:54.439071  8670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:54.440078  8670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:54.441280  8670 hybrid_clock.cc:648] HybridClock initialized: now 1714702374441231 us; error 42 us; skew 500 ppm
I20240503 02:12:54.441627  8670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:54.487789  8670 webserver.cc:460] Webserver started at http://127.0.145.7:45823/ using document root <none> and password file <none>
I20240503 02:12:54.488679  8670 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:54.488811  8670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:54.489171  8670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:54.492051  8670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "d826d4dae2aa4e05b2f64c9b17a1126f"
format_stamp: "Formatted at 2024-05-03 02:12:54 on dist-test-slave-gq9h"
I20240503 02:12:54.492851  8670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "d826d4dae2aa4e05b2f64c9b17a1126f"
format_stamp: "Formatted at 2024-05-03 02:12:54 on dist-test-slave-gq9h"
I20240503 02:12:54.498354  8670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20240503 02:12:54.501911  8684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:54.503464  8670 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:12:54.503661  8670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d826d4dae2aa4e05b2f64c9b17a1126f"
format_stamp: "Formatted at 2024-05-03 02:12:54 on dist-test-slave-gq9h"
I20240503 02:12:54.503897  8670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:54.522477  8670 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:54.523403  8670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:54.523674  8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:54.524654  8670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:54.526211  8670 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:54.526305  8670 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:54.526438  8670 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:54.526508  8670 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:54.562165  8670 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.7:39415
I20240503 02:12:54.563575  8670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:12:54.562208  8796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.7:39415 every 8 connection(s)
I20240503 02:12:54.571656   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 8670
I20240503 02:12:54.571827   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:12:54.575413   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.8:0
--local_ip_for_outbound_sockets=127.0.145.8
--webserver_interface=127.0.145.8
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--builtin_ntp_servers=127.0.145.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:54.577016  8797 heartbeater.cc:347] Connected to a master server at 127.0.145.62:36865
I20240503 02:12:54.577293  8797 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:54.578233  8797 heartbeater.cc:511] Master 127.0.145.62:36865 requested a full tablet report, sending...
I20240503 02:12:54.579667  8537 heartbeater.cc:503] Master 127.0.145.62:36865 was elected leader, sending a full tablet report...
I20240503 02:12:54.579761  7830 ts_manager.cc:194] Registered new tserver with Master: d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
I20240503 02:12:54.580669  7830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.7:48659
W20240503 02:12:54.927906  8801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:54.928229  8801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:54.928393  8801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:54.936235  8801 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:54.936393  8801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:54.936573  8801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.8
I20240503 02:12:54.946115  8801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.145.8
--webserver_port=0
--tserver_master_addrs=127.0.145.62:36865
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.8
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:54.947569  8801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:54.949350  8801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:54.962853  8806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:54.963714  8807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:54.964488  8809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:54.969552  8801 server_base.cc:1034] running on GCE node
I20240503 02:12:54.970213  8801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:54.971248  8801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:54.972440  8801 hybrid_clock.cc:648] HybridClock initialized: now 1714702374972394 us; error 39 us; skew 500 ppm
I20240503 02:12:54.972728  8801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:54.974967  8801 webserver.cc:460] Webserver started at http://127.0.145.8:46757/ using document root <none> and password file <none>
I20240503 02:12:54.975667  8801 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:54.975766  8801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:54.976054  8801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:54.978803  8801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4"
format_stamp: "Formatted at 2024-05-03 02:12:54 on dist-test-slave-gq9h"
I20240503 02:12:54.979575  8801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4"
format_stamp: "Formatted at 2024-05-03 02:12:54 on dist-test-slave-gq9h"
I20240503 02:12:54.985414  8801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:12:54.989604  8815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:54.991247  8801 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240503 02:12:54.991437  8801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4"
format_stamp: "Formatted at 2024-05-03 02:12:54 on dist-test-slave-gq9h"
I20240503 02:12:54.991657  8801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:55.016865  8801 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:55.017776  8801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:55.018064  8801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:55.019150  8801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:55.020694  8801 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:55.020807  8801 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:55.020941  8801 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:55.021016  8801 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:55.055321  8667 heartbeater.cc:503] Master 127.0.145.62:36865 was elected leader, sending a full tablet report...
I20240503 02:12:55.057650  8801 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.8:38885
I20240503 02:12:55.057742  8927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.8:38885 every 8 connection(s)
I20240503 02:12:55.059139  8801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:12:55.067987   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 8801
I20240503 02:12:55.068176   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:12:55.072438  8928 heartbeater.cc:347] Connected to a master server at 127.0.145.62:36865
I20240503 02:12:55.072763  8928 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:55.073624  8928 heartbeater.cc:511] Master 127.0.145.62:36865 requested a full tablet report, sending...
I20240503 02:12:55.075058  7829 ts_manager.cc:194] Registered new tserver with Master: ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
I20240503 02:12:55.076202  7829 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.8:51145
I20240503 02:12:55.084573   580 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:12:55.102094   580 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:12:55.102216   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 8410
I20240503 02:12:55.111173  8532 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:55.193712   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 8410
I20240503 02:12:55.210568   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 8540
I20240503 02:12:55.219072  8662 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:55.299266   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 8540
I20240503 02:12:55.316812   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 8670
I20240503 02:12:55.325260  8792 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:55.407322   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 8670
I20240503 02:12:55.423704   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 8801
I20240503 02:12:55.432287  8923 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:55.513816   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 8801
I20240503 02:12:59.283676  7829 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50564:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:12:59.321700  8082 tablet_service.cc:1467] Processing CreateTablet for tablet 80b4e6aa20f64473be8bcbf47baf66e4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.321700  8341 tablet_service.cc:1467] Processing CreateTablet for tablet f14607425da84bb4b162a5789daa844c (DEFAULT_TABLE table=rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.323292  8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14607425da84bb4b162a5789daa844c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.322836  8081 tablet_service.cc:1467] Processing CreateTablet for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac (DEFAULT_TABLE table=rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.324256  8081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.324209  8211 tablet_service.cc:1467] Processing CreateTablet for tablet f14607425da84bb4b162a5789daa844c (DEFAULT_TABLE table=rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.325706  8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14607425da84bb4b162a5789daa844c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.325771  8342 tablet_service.cc:1467] Processing CreateTablet for tablet 80b4e6aa20f64473be8bcbf47baf66e4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.326331  8342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b4e6aa20f64473be8bcbf47baf66e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.330929  8212 tablet_service.cc:1467] Processing CreateTablet for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac (DEFAULT_TABLE table=rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.331499  8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.331521  8082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b4e6aa20f64473be8bcbf47baf66e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.336169  7950 tablet_service.cc:1467] Processing CreateTablet for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac (DEFAULT_TABLE table=rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.337782  7950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.338830  7951 tablet_service.cc:1467] Processing CreateTablet for tablet 80b4e6aa20f64473be8bcbf47baf66e4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.339426  7951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b4e6aa20f64473be8bcbf47baf66e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.339490  7949 tablet_service.cc:1467] Processing CreateTablet for tablet f14607425da84bb4b162a5789daa844c (DEFAULT_TABLE table=rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.339964  7949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14607425da84bb4b162a5789daa844c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.343031  8957 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:12:59.346812  8957 tablet_bootstrap.cc:654] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.348279  8957 log.cc:824] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:59.354779  8957 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:12:59.355244  8957 ts_tablet_manager.cc:1367] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.012s	user 0.002s	sys 0.009s
I20240503 02:12:59.357981  8958 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:12:59.364951  8958 tablet_bootstrap.cc:654] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.366536  8958 log.cc:824] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:59.371840  8961 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:12:59.372067  8957 raft_consensus.cc:348] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.372768  8957 raft_consensus.cc:374] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.372967  8957 raft_consensus.cc:729] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.374007  8957 consensus_queue.cc:260] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.376716  8958 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:12:59.377112  8958 ts_tablet_manager.cc:1367] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.019s	user 0.006s	sys 0.008s
I20240503 02:12:59.377288  8961 tablet_bootstrap.cc:654] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.378693  8961 log.cc:824] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:59.379595  8960 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:12:59.383500  8960 tablet_bootstrap.cc:654] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.384922  8960 log.cc:824] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:59.386626  8958 raft_consensus.cc:348] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.387375  8958 raft_consensus.cc:374] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.387571  8958 raft_consensus.cc:729] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.389250  8958 consensus_queue.cc:260] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.389947  8957 ts_tablet_manager.cc:1398] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.034s	user 0.008s	sys 0.017s
I20240503 02:12:59.390425  8957 tablet_bootstrap.cc:492] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:12:59.392195  8957 tablet_bootstrap.cc:654] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.392354  8960 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:12:59.392699  8960 ts_tablet_manager.cc:1367] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.013s	user 0.007s	sys 0.004s
I20240503 02:12:59.401376  8961 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:12:59.401945  8961 ts_tablet_manager.cc:1367] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.030s	user 0.006s	sys 0.011s
I20240503 02:12:59.404477  8958 ts_tablet_manager.cc:1398] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.027s	user 0.012s	sys 0.010s
I20240503 02:12:59.404899  8958 tablet_bootstrap.cc:492] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:12:59.404954  8960 raft_consensus.cc:348] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.405474  8960 raft_consensus.cc:374] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.405589  8960 raft_consensus.cc:729] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.406314  8960 consensus_queue.cc:260] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.406733  8958 tablet_bootstrap.cc:654] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.407603  8957 tablet_bootstrap.cc:492] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:12:59.407778  8957 ts_tablet_manager.cc:1367] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.017s	user 0.003s	sys 0.010s
I20240503 02:12:59.408896  8957 raft_consensus.cc:348] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:12:59.409103  8957 raft_consensus.cc:374] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.409210  8957 raft_consensus.cc:729] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.409464  8957 consensus_queue.cc:260] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:12:59.410002  8957 ts_tablet_manager.cc:1398] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:59.418385  8961 raft_consensus.cc:348] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.419697  8960 ts_tablet_manager.cc:1398] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.027s	user 0.018s	sys 0.004s
I20240503 02:12:59.419868  8961 raft_consensus.cc:374] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.420050  8958 tablet_bootstrap.cc:492] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:12:59.420063  8961 raft_consensus.cc:729] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.420233  8960 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:12:59.420377  8958 ts_tablet_manager.cc:1367] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.016s	user 0.000s	sys 0.015s
I20240503 02:12:59.420900  8961 consensus_queue.cc:260] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.421658  8958 raft_consensus.cc:348] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:12:59.421949  8958 raft_consensus.cc:374] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.422089  8958 raft_consensus.cc:729] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.422219  8960 tablet_bootstrap.cc:654] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.422438  8958 consensus_queue.cc:260] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:12:59.423228  8958 ts_tablet_manager.cc:1398] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.001s
I20240503 02:12:59.424823  8960 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:12:59.425029  8960 ts_tablet_manager.cc:1367] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.002s
I20240503 02:12:59.426237  8961 ts_tablet_manager.cc:1398] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.024s	user 0.015s	sys 0.002s
I20240503 02:12:59.426127  8960 raft_consensus.cc:348] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.426352  8960 raft_consensus.cc:374] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.426491  8960 raft_consensus.cc:729] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.426791  8960 consensus_queue.cc:260] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.426995  8961 tablet_bootstrap.cc:492] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:12:59.427404  8960 ts_tablet_manager.cc:1398] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:59.428812  8961 tablet_bootstrap.cc:654] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.431303  8961 tablet_bootstrap.cc:492] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:12:59.431468  8961 ts_tablet_manager.cc:1367] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.001s
I20240503 02:12:59.432307  8961 raft_consensus.cc:348] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:12:59.432475  8961 raft_consensus.cc:374] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.432576  8961 raft_consensus.cc:729] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.432788  8961 consensus_queue.cc:260] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:12:59.433241  8961 ts_tablet_manager.cc:1398] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:59.433501  8961 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:12:59.434839  8961 tablet_bootstrap.cc:654] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.436673  8961 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:12:59.436820  8961 ts_tablet_manager.cc:1367] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 02:12:59.437430  8961 raft_consensus.cc:348] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.437611  8961 raft_consensus.cc:374] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.437683  8961 raft_consensus.cc:729] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.437893  8961 consensus_queue.cc:260] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.438354  8961 ts_tablet_manager.cc:1398] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:12:59.512809  8966 raft_consensus.cc:482] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:59.513075  8966 raft_consensus.cc:504] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.515008  8966 leader_election.cc:288] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:12:59.522342  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d69c71d9dfd47a1a2429e7caf5e33ac" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:12:59.522388  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d69c71d9dfd47a1a2429e7caf5e33ac" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:12:59.522864  7971 raft_consensus.cc:2454] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:12:59.522863  8102 raft_consensus.cc:2454] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:12:59.523486  8167 leader_election.cc:302] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:12:59.523873  8966 raft_consensus.cc:2781] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:59.524036  8966 raft_consensus.cc:482] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:59.524138  8966 raft_consensus.cc:3037] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.526531  8966 raft_consensus.cc:504] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.527709  8966 leader_election.cc:288] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Requested vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:12:59.528645  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d69c71d9dfd47a1a2429e7caf5e33ac" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:12:59.528774  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d69c71d9dfd47a1a2429e7caf5e33ac" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:12:59.529052  7971 raft_consensus.cc:3037] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.529177  8102 raft_consensus.cc:3037] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.532084  8102 raft_consensus.cc:2454] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:12:59.532083  7971 raft_consensus.cc:2454] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:12:59.532549  8165 leader_election.cc:302] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:12:59.532861  8966 raft_consensus.cc:2781] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:59.533272  8966 raft_consensus.cc:686] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Becoming Leader. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Running, Role: LEADER
I20240503 02:12:59.533787  8966 consensus_queue.cc:237] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.538558  7828 catalog_manager.cc:5484] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3). New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:59.558979  8408 tablet.cc:2369] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:59.572566  8963 raft_consensus.cc:482] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:59.572875  8963 raft_consensus.cc:504] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.574687  8963 leader_election.cc:288] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:12:59.583655  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b4e6aa20f64473be8bcbf47baf66e4" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:12:59.583962  7971 raft_consensus.cc:2454] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:12:59.584194  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b4e6aa20f64473be8bcbf47baf66e4" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:12:59.584450  8102 raft_consensus.cc:2454] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:12:59.584604  8295 leader_election.cc:302] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:12:59.585073  8963 raft_consensus.cc:2781] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:59.585224  8963 raft_consensus.cc:482] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:59.585330  8963 raft_consensus.cc:3037] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.587467  8963 raft_consensus.cc:504] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.588132  8963 leader_election.cc:288] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Requested vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:12:59.588629  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b4e6aa20f64473be8bcbf47baf66e4" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:12:59.588706  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b4e6aa20f64473be8bcbf47baf66e4" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:12:59.588874  7971 raft_consensus.cc:3037] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.588937  8102 raft_consensus.cc:3037] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.591781  7971 raft_consensus.cc:2454] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:12:59.591781  8102 raft_consensus.cc:2454] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:12:59.592250  8295 leader_election.cc:302] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:12:59.592584  8963 raft_consensus.cc:2781] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:59.593091  8963 raft_consensus.cc:686] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Running, Role: LEADER
W20240503 02:12:59.593111  8278 tablet.cc:2369] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:59.593672  8963 consensus_queue.cc:237] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:12:59.598141  7828 catalog_manager.cc:5484] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d7c037d50c241b484ee843b06313a64 (127.0.145.4). New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:59.616523  8148 tablet.cc:2369] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:12:59.616606  8017 tablet.cc:2369] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:59.630631  8966 raft_consensus.cc:482] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:59.630888  8966 raft_consensus.cc:504] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:12:59.631553  8966 leader_election.cc:288] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:12:59.632198  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14607425da84bb4b162a5789daa844c" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:12:59.632436  7971 raft_consensus.cc:2454] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:12:59.632872  8165 leader_election.cc:302] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:12:59.633162  8966 raft_consensus.cc:2781] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:59.633244  8966 raft_consensus.cc:482] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:59.633345  8966 raft_consensus.cc:3037] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.635587  8966 raft_consensus.cc:504] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:12:59.636207  8966 leader_election.cc:288] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Requested vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:12:59.636727  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14607425da84bb4b162a5789daa844c" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:12:59.636915  7971 raft_consensus.cc:3037] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.638094  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14607425da84bb4b162a5789daa844c" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:12:59.638119  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14607425da84bb4b162a5789daa844c" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:12:59.638587  8362 raft_consensus.cc:2454] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:12:59.639495  7971 raft_consensus.cc:2454] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:12:59.639906  8165 leader_election.cc:302] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, d986fcfb926e47baa7d2d5b99f697334; no voters: 8d7c037d50c241b484ee843b06313a64
I20240503 02:12:59.640239  8966 raft_consensus.cc:2781] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:59.640424  8966 raft_consensus.cc:686] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Becoming Leader. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Running, Role: LEADER
I20240503 02:12:59.640694  8966 consensus_queue.cc:237] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:12:59.643460  7828 catalog_manager.cc:5484] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3). New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:59.943687  8966 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:59.952337  8966 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:00.030176  8983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:00.030555  8983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:00.038327  8983 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:00.065373  8963 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.065934  8980 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 02:13:00.157513  8966 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.160534  8361 raft_consensus.cc:3037] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.165364  8966 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 342.422 ms
  time per row: 0.171211 ms
I20240503 02:13:00.543952  7828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50564:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:13:00.553007  8082 tablet_service.cc:1467] Processing CreateTablet for tablet 32e527fa9b2f441cb5668435681b4ff6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d4270f8900eb45459ee1e49c30ff6cff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.553260  8342 tablet_service.cc:1467] Processing CreateTablet for tablet 32e527fa9b2f441cb5668435681b4ff6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d4270f8900eb45459ee1e49c30ff6cff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.553297  8212 tablet_service.cc:1467] Processing CreateTablet for tablet 32e527fa9b2f441cb5668435681b4ff6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d4270f8900eb45459ee1e49c30ff6cff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.553520  8082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e527fa9b2f441cb5668435681b4ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.553674  8342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e527fa9b2f441cb5668435681b4ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.553750  8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e527fa9b2f441cb5668435681b4ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.556381  8080 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8affdde4d8417f95f0585dd01e7d59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d4270f8900eb45459ee1e49c30ff6cff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.556460  8341 tablet_service.cc:1467] Processing CreateTablet for tablet 3dbfc7f12fc142f98b1d1c026fd49f5a (DEFAULT_TABLE table=rebalance_test_table_1 [id=d4270f8900eb45459ee1e49c30ff6cff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.556880  8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dbfc7f12fc142f98b1d1c026fd49f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.556888  8080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8affdde4d8417f95f0585dd01e7d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.557963  8081 tablet_service.cc:1467] Processing CreateTablet for tablet 3dbfc7f12fc142f98b1d1c026fd49f5a (DEFAULT_TABLE table=rebalance_test_table_1 [id=d4270f8900eb45459ee1e49c30ff6cff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.558395  8081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dbfc7f12fc142f98b1d1c026fd49f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.562058  8211 tablet_service.cc:1467] Processing CreateTablet for tablet 3dbfc7f12fc142f98b1d1c026fd49f5a (DEFAULT_TABLE table=rebalance_test_table_1 [id=d4270f8900eb45459ee1e49c30ff6cff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.562052  7950 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8affdde4d8417f95f0585dd01e7d59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d4270f8900eb45459ee1e49c30ff6cff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.562533  8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dbfc7f12fc142f98b1d1c026fd49f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.562640  7950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8affdde4d8417f95f0585dd01e7d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.567513  8210 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8affdde4d8417f95f0585dd01e7d59 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d4270f8900eb45459ee1e49c30ff6cff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.568123  8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8affdde4d8417f95f0585dd01e7d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.568989  9044 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:00.571082  9044 tablet_bootstrap.cc:654] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.572388  9045 tablet_bootstrap.cc:492] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:00.574363  9045 tablet_bootstrap.cc:654] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.575963  9047 tablet_bootstrap.cc:492] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:00.582413  9045 tablet_bootstrap.cc:492] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:00.582967  9044 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:00.583189  9044 ts_tablet_manager.cc:1367] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.014s	user 0.010s	sys 0.000s
I20240503 02:13:00.584056  9044 raft_consensus.cc:348] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.584283  9044 raft_consensus.cc:374] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.584378  9044 raft_consensus.cc:729] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.584615  9044 consensus_queue.cc:260] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.585150  9044 ts_tablet_manager.cc:1398] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:13:00.582598  9045 ts_tablet_manager.cc:1367] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.001s
I20240503 02:13:00.585904  9045 raft_consensus.cc:348] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.586079  9045 raft_consensus.cc:374] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.586190  9045 raft_consensus.cc:729] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.586468  9045 consensus_queue.cc:260] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.587049  9044 tablet_bootstrap.cc:492] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:00.588205  9047 tablet_bootstrap.cc:654] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.588716  9044 tablet_bootstrap.cc:654] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.590039  9045 ts_tablet_manager.cc:1398] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.001s
I20240503 02:13:00.590425  9045 tablet_bootstrap.cc:492] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:00.592162  9045 tablet_bootstrap.cc:654] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.594306  9045 tablet_bootstrap.cc:492] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:00.594481  9045 ts_tablet_manager.cc:1367] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:13:00.595146  9045 raft_consensus.cc:348] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:00.595345  9045 raft_consensus.cc:374] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.595428  9045 raft_consensus.cc:729] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.595724  9045 consensus_queue.cc:260] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:00.596253  9045 ts_tablet_manager.cc:1398] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:13:00.596522  9045 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:00.598125  9045 tablet_bootstrap.cc:654] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.601992  9047 tablet_bootstrap.cc:492] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:00.602208  9047 ts_tablet_manager.cc:1367] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.026s	user 0.006s	sys 0.000s
I20240503 02:13:00.603499  9047 raft_consensus.cc:348] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.603829  9047 raft_consensus.cc:374] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.603953  9047 raft_consensus.cc:729] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.604311  9047 consensus_queue.cc:260] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.605252  9047 ts_tablet_manager.cc:1398] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:13:00.605660  9047 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:00.607678  9047 tablet_bootstrap.cc:654] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.610714  9044 tablet_bootstrap.cc:492] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:00.610905  9044 ts_tablet_manager.cc:1367] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.024s	user 0.005s	sys 0.004s
I20240503 02:13:00.611687  9045 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:00.611693  9044 raft_consensus.cc:348] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.611933  9044 raft_consensus.cc:374] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.612004  9044 raft_consensus.cc:729] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.612290  9044 consensus_queue.cc:260] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.612648  9045 ts_tablet_manager.cc:1367] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.016s	user 0.003s	sys 0.004s
I20240503 02:13:00.612748  9047 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:00.612938  9047 ts_tablet_manager.cc:1367] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.004s
I20240503 02:13:00.613639  9044 ts_tablet_manager.cc:1398] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:13:00.614369  9045 raft_consensus.cc:348] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.614562  8980 raft_consensus.cc:482] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:00.614745  9045 raft_consensus.cc:374] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.614758  8980 raft_consensus.cc:504] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.614912  9045 raft_consensus.cc:729] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.615296  9045 consensus_queue.cc:260] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.615582  8980 leader_election.cc:288] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:00.616031  9045 ts_tablet_manager.cc:1398] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240503 02:13:00.616283  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dbfc7f12fc142f98b1d1c026fd49f5a" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:00.616490  8101 raft_consensus.cc:2454] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:13:00.617589  8297 leader_election.cc:302] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:00.617933  8980 raft_consensus.cc:2781] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:00.618043  8980 raft_consensus.cc:482] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:00.618005  9047 raft_consensus.cc:348] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.618166  8980 raft_consensus.cc:3037] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.618264  9047 raft_consensus.cc:374] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.618355  9047 raft_consensus.cc:729] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.618710  9047 consensus_queue.cc:260] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.621626  9047 ts_tablet_manager.cc:1398] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.008s	user 0.000s	sys 0.002s
I20240503 02:13:00.621651  8980 raft_consensus.cc:504] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.623261  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dbfc7f12fc142f98b1d1c026fd49f5a" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:00.623461  8101 raft_consensus.cc:3037] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.624099  9047 tablet_bootstrap.cc:492] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:00.626348  8101 raft_consensus.cc:2454] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:00.608449  9049 tablet_bootstrap.cc:492] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:00.626771  9047 tablet_bootstrap.cc:654] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.626823  8297 leader_election.cc:302] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:00.626858  8980 leader_election.cc:288] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Requested vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:00.627158  8980 raft_consensus.cc:2781] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:00.627363  8980 raft_consensus.cc:686] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Running, Role: LEADER
I20240503 02:13:00.627674  8980 consensus_queue.cc:237] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.628651  9049 tablet_bootstrap.cc:654] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.636595  9049 tablet_bootstrap.cc:492] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:00.636790  9049 ts_tablet_manager.cc:1367] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.028s	user 0.004s	sys 0.000s
I20240503 02:13:00.636698  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dbfc7f12fc142f98b1d1c026fd49f5a" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:00.636690  7822 catalog_manager.cc:5484] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d7c037d50c241b484ee843b06313a64 (127.0.145.4). New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:00.637140  8231 raft_consensus.cc:3037] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.637766  9049 raft_consensus.cc:348] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:00.637988  9049 raft_consensus.cc:374] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.638078  9049 raft_consensus.cc:729] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.638375  9049 consensus_queue.cc:260] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:00.638257  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dbfc7f12fc142f98b1d1c026fd49f5a" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:00.639065  9049 ts_tablet_manager.cc:1398] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:13:00.640569  8231 raft_consensus.cc:2454] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:00.642755  9047 tablet_bootstrap.cc:492] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:00.642949  9047 ts_tablet_manager.cc:1367] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.021s	user 0.002s	sys 0.010s
I20240503 02:13:00.643801  9047 raft_consensus.cc:348] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:00.644001  9047 raft_consensus.cc:374] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.644088  9047 raft_consensus.cc:729] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.644342  9047 consensus_queue.cc:260] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:00.645056  9047 ts_tablet_manager.cc:1398] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:13:00.759562  9050 raft_consensus.cc:482] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:00.759788  9050 raft_consensus.cc:504] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:00.761425  9050 leader_election.cc:288] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:00.767400  8966 raft_consensus.cc:482] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:00.767621  8966 raft_consensus.cc:504] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.768471  8966 leader_election.cc:288] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:00.769266  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8affdde4d8417f95f0585dd01e7d59" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:00.769296  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e527fa9b2f441cb5668435681b4ff6" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:00.769523  8231 raft_consensus.cc:2454] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:00.769516  8361 raft_consensus.cc:2454] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:13:00.769909  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e527fa9b2f441cb5668435681b4ff6" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:00.770005  8035 leader_election.cc:302] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:00.770144  8101 raft_consensus.cc:2454] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:13:00.770439  9050 raft_consensus.cc:2781] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:00.770558  9050 raft_consensus.cc:482] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:00.770680  9050 raft_consensus.cc:3037] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.770812  8168 leader_election.cc:302] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64; no voters: 
I20240503 02:13:00.770812  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8affdde4d8417f95f0585dd01e7d59" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:13:00.771016  7969 raft_consensus.cc:2454] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:00.771111  8966 raft_consensus.cc:2781] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:00.771224  8966 raft_consensus.cc:482] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:00.771308  8966 raft_consensus.cc:3037] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.773638  9050 raft_consensus.cc:504] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:00.774040  8966 raft_consensus.cc:504] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.774401  9050 leader_election.cc:288] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Requested vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:00.774878  8966 leader_election.cc:288] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:00.775451  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e527fa9b2f441cb5668435681b4ff6" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:00.775462  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8affdde4d8417f95f0585dd01e7d59" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:00.775691  8231 raft_consensus.cc:3037] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.775619  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e527fa9b2f441cb5668435681b4ff6" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:00.775689  8361 raft_consensus.cc:3037] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.775831  8101 raft_consensus.cc:3037] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.777798  8231 raft_consensus.cc:2454] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:00.778234  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8affdde4d8417f95f0585dd01e7d59" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:00.778311  8035 leader_election.cc:302] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:00.778474  7969 raft_consensus.cc:3037] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.778654  8101 raft_consensus.cc:2454] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:13:00.778677  8361 raft_consensus.cc:2454] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:13:00.778949  9050 raft_consensus.cc:2781] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:00.779444  9050 raft_consensus.cc:686] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Becoming Leader. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:13:00.779469  8167 leader_election.cc:302] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:00.779783  8966 raft_consensus.cc:2781] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:00.779911  8966 raft_consensus.cc:686] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Becoming Leader. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Running, Role: LEADER
I20240503 02:13:00.780187  8966 consensus_queue.cc:237] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:00.780458  9050 consensus_queue.cc:237] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:00.781356  7969 raft_consensus.cc:2454] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:00.783947  7822 catalog_manager.cc:5484] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3). New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:00.784981  7828 catalog_manager.cc:5484] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2). New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.077924  8980 consensus_queue.cc:1035] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.084242  8980 consensus_queue.cc:1035] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:01.182780  9066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:01.183115  9066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:01.190778  9066 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:13:01.232539  9050 consensus_queue.cc:1035] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.241237  9050 consensus_queue.cc:1035] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.285478  8987 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.291092  8966 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 246.239 ms
  time per row: 0.12312 ms
I20240503 02:13:01.604379  7822 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50564:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:13:01.614825  7950 tablet_service.cc:1467] Processing CreateTablet for tablet cab70b62bdc747a9b653bffde680f37e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.615203  8341 tablet_service.cc:1467] Processing CreateTablet for tablet cab70b62bdc747a9b653bffde680f37e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.615365  7950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cab70b62bdc747a9b653bffde680f37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.615636  8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cab70b62bdc747a9b653bffde680f37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.622203  8342 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba1f505010b403c91d3870cee19952a (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.623162  8082 tablet_service.cc:1467] Processing CreateTablet for tablet cab70b62bdc747a9b653bffde680f37e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.623687  8080 tablet_service.cc:1467] Processing CreateTablet for tablet 0f44b2b53de84a87a8757c0d00bfed37 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.624740  8210 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba1f505010b403c91d3870cee19952a (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.625051  8082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cab70b62bdc747a9b653bffde680f37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.625046  7949 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba1f505010b403c91d3870cee19952a (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.625196  8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba1f505010b403c91d3870cee19952a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.625480  7949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba1f505010b403c91d3870cee19952a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.627115  7951 tablet_service.cc:1467] Processing CreateTablet for tablet 0f44b2b53de84a87a8757c0d00bfed37 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.627506  7951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f44b2b53de84a87a8757c0d00bfed37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.630820  8080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f44b2b53de84a87a8757c0d00bfed37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.639132  9133 tablet_bootstrap.cc:492] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:01.639371  8342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba1f505010b403c91d3870cee19952a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.641070  9133 tablet_bootstrap.cc:654] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.643252  8340 tablet_service.cc:1467] Processing CreateTablet for tablet 0f44b2b53de84a87a8757c0d00bfed37 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.643581  9132 tablet_bootstrap.cc:492] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:01.643738  8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f44b2b53de84a87a8757c0d00bfed37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.645505  9132 tablet_bootstrap.cc:654] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.648738  9133 tablet_bootstrap.cc:492] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:01.648970  9133 ts_tablet_manager.cc:1367] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.000s
I20240503 02:13:01.650113  9133 raft_consensus.cc:348] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.650406  9133 raft_consensus.cc:374] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.650525  9133 raft_consensus.cc:729] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.651137  9134 tablet_bootstrap.cc:492] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:01.651247  9137 tablet_bootstrap.cc:492] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:01.651129  9133 consensus_queue.cc:260] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.651894  9133 ts_tablet_manager.cc:1398] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:13:01.652184  9133 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:01.653656  9132 tablet_bootstrap.cc:492] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:01.653856  9132 ts_tablet_manager.cc:1367] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.000s
I20240503 02:13:01.653926  9133 tablet_bootstrap.cc:654] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.654094  9134 tablet_bootstrap.cc:654] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.654886  9132 raft_consensus.cc:348] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:01.655203  9132 raft_consensus.cc:374] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.655318  9132 raft_consensus.cc:729] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.655711  9132 consensus_queue.cc:260] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:01.656636  9132 ts_tablet_manager.cc:1398] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:13:01.657997  9137 tablet_bootstrap.cc:654] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.658821  9133 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:01.659024  9133 ts_tablet_manager.cc:1367] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240503 02:13:01.659659  9133 raft_consensus.cc:348] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.659916  9133 raft_consensus.cc:374] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.659727  9134 tablet_bootstrap.cc:492] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:01.660012  9133 raft_consensus.cc:729] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.660243  9134 ts_tablet_manager.cc:1367] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.001s
I20240503 02:13:01.660282  9133 consensus_queue.cc:260] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.660854  9133 ts_tablet_manager.cc:1398] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:13:01.661295  9133 tablet_bootstrap.cc:492] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:01.662034  9134 raft_consensus.cc:348] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.662468  9134 raft_consensus.cc:374] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.662671  9134 raft_consensus.cc:729] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.663156  9133 tablet_bootstrap.cc:654] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.663086  9134 consensus_queue.cc:260] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.664755  9134 ts_tablet_manager.cc:1398] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240503 02:13:01.665670  9134 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:01.666252  9133 tablet_bootstrap.cc:492] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:01.666409  9133 ts_tablet_manager.cc:1367] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:13:01.667277  9133 raft_consensus.cc:348] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:01.667227  9137 tablet_bootstrap.cc:492] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:01.667485  9133 raft_consensus.cc:374] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.667574  9133 raft_consensus.cc:729] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.667627  9134 tablet_bootstrap.cc:654] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.667644  9137 ts_tablet_manager.cc:1367] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.017s	user 0.002s	sys 0.007s
I20240503 02:13:01.667843  9133 consensus_queue.cc:260] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:01.668395  9133 ts_tablet_manager.cc:1398] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:13:01.668470  9137 raft_consensus.cc:348] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.668692  9137 raft_consensus.cc:374] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.668784  9137 raft_consensus.cc:729] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.669057  9137 consensus_queue.cc:260] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.669720  9134 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:01.669764  9137 ts_tablet_manager.cc:1398] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:13:01.669890  9134 ts_tablet_manager.cc:1367] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:13:01.670218  9137 tablet_bootstrap.cc:492] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:01.670670  9134 raft_consensus.cc:348] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.670898  9134 raft_consensus.cc:374] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.671000  9134 raft_consensus.cc:729] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.671259  9134 consensus_queue.cc:260] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.671798  9134 ts_tablet_manager.cc:1398] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:13:01.672761  9137 tablet_bootstrap.cc:654] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.674917  9137 tablet_bootstrap.cc:492] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:01.675087  9137 ts_tablet_manager.cc:1367] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240503 02:13:01.675756  9137 raft_consensus.cc:348] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:01.675987  9137 raft_consensus.cc:374] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.676076  9137 raft_consensus.cc:729] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.676304  9137 consensus_queue.cc:260] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:01.676708  9137 ts_tablet_manager.cc:1398] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:13:01.677028  9137 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:01.678174  9137 tablet_bootstrap.cc:654] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.679978  9137 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:01.680119  9137 ts_tablet_manager.cc:1367] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:13:01.680716  9137 raft_consensus.cc:348] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.680896  9137 raft_consensus.cc:374] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.680997  9137 raft_consensus.cc:729] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.681210  9137 consensus_queue.cc:260] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.681602  9137 ts_tablet_manager.cc:1398] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:13:01.746455  9088 raft_consensus.cc:482] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.746695  9088 raft_consensus.cc:504] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.748415  9088 leader_election.cc:288] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:01.757038  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab70b62bdc747a9b653bffde680f37e" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:01.757026  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab70b62bdc747a9b653bffde680f37e" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:01.757329  8361 raft_consensus.cc:2454] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 0.
I20240503 02:13:01.757306  8102 raft_consensus.cc:2454] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 0.
I20240503 02:13:01.757845  7907 leader_election.cc:302] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:01.758293  9088 raft_consensus.cc:2781] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.758430  9088 raft_consensus.cc:482] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.758560  9088 raft_consensus.cc:3037] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.760615  9088 raft_consensus.cc:504] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.761287  9088 leader_election.cc:288] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 election: Requested vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:01.761813  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab70b62bdc747a9b653bffde680f37e" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:01.761857  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cab70b62bdc747a9b653bffde680f37e" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:01.762028  8361 raft_consensus.cc:3037] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.762043  8102 raft_consensus.cc:3037] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.763959  8361 raft_consensus.cc:2454] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 1.
I20240503 02:13:01.763984  8102 raft_consensus.cc:2454] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 1.
I20240503 02:13:01.764469  7906 leader_election.cc:302] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:01.764796  9088 raft_consensus.cc:2781] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.765275  9088 raft_consensus.cc:686] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Becoming Leader. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Running, Role: LEADER
I20240503 02:13:01.765780  9088 consensus_queue.cc:237] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.769348  7822 catalog_manager.cc:5484] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: term changed from 0 to 1, leader changed from <none> to d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.800454  8980 raft_consensus.cc:482] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.800662  8980 raft_consensus.cc:504] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:01.801296  8980 leader_election.cc:288] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:01.801939  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1f505010b403c91d3870cee19952a" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:01.801986  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1f505010b403c91d3870cee19952a" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:13:01.802174  8231 raft_consensus.cc:2454] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:13:01.802251  7969 raft_consensus.cc:2454] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:13:01.802660  8295 leader_election.cc:302] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64; no voters: 
I20240503 02:13:01.802963  8980 raft_consensus.cc:2781] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.803083  8980 raft_consensus.cc:482] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.803176  8980 raft_consensus.cc:3037] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.805167  8980 raft_consensus.cc:504] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:01.805718  8980 leader_election.cc:288] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Requested vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:01.806380  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1f505010b403c91d3870cee19952a" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:01.806378  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1f505010b403c91d3870cee19952a" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:01.806650  8231 raft_consensus.cc:3037] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.806655  7969 raft_consensus.cc:3037] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.809597  7969 raft_consensus.cc:2454] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:01.809612  8231 raft_consensus.cc:2454] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:01.810071  8295 leader_election.cc:302] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64; no voters: 
I20240503 02:13:01.810353  8980 raft_consensus.cc:2781] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.810541  8980 raft_consensus.cc:686] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Running, Role: LEADER
I20240503 02:13:01.810856  8980 consensus_queue.cc:237] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:01.813529  7822 catalog_manager.cc:5484] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d7c037d50c241b484ee843b06313a64 (127.0.145.4). New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.823935  9050 raft_consensus.cc:482] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.824172  9050 raft_consensus.cc:504] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.824890  9050 leader_election.cc:288] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:01.825489  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f44b2b53de84a87a8757c0d00bfed37" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:13:01.825752  7969 raft_consensus.cc:2454] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:01.826206  8035 leader_election.cc:302] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:01.826489  9050 raft_consensus.cc:2781] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.826645  9050 raft_consensus.cc:482] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.826773  9050 raft_consensus.cc:3037] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.828831  9050 raft_consensus.cc:504] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.829478  9050 leader_election.cc:288] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Requested vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:01.830014  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f44b2b53de84a87a8757c0d00bfed37" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:01.830255  7969 raft_consensus.cc:3037] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.831609  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f44b2b53de84a87a8757c0d00bfed37" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:01.831734  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f44b2b53de84a87a8757c0d00bfed37" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:01.831840  8361 raft_consensus.cc:2454] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:01.832587  7969 raft_consensus.cc:2454] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:01.832978  8035 leader_election.cc:302] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d986fcfb926e47baa7d2d5b99f697334; no voters: 8d7c037d50c241b484ee843b06313a64
I20240503 02:13:01.833272  9050 raft_consensus.cc:2781] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.833433  9050 raft_consensus.cc:686] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Becoming Leader. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:13:01.834000  9050 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:01.837045  7822 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2). New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } }
W20240503 02:13:02.229413  9152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:02.229830  9152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:02.237972  9152 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:02.245824  9088 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.254176  9088 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 02:13:02.301510  9121 consensus_queue.cc:1035] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.306408  9121 consensus_queue.cc:1035] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.317780  9050 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.339712  8362 raft_consensus.cc:3037] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.344399  9050 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 248.384 ms
  time per row: 0.124192 ms
I20240503 02:13:02.652326  7822 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50564:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:13:02.661901  8210 tablet_service.cc:1467] Processing CreateTablet for tablet 007b59de142c4b95966494ba097b1a09 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9af1e1dd28c144d9b52b73387f91d232]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.662204  8082 tablet_service.cc:1467] Processing CreateTablet for tablet 007b59de142c4b95966494ba097b1a09 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9af1e1dd28c144d9b52b73387f91d232]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.662446  8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007b59de142c4b95966494ba097b1a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.662649  8082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007b59de142c4b95966494ba097b1a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.664960  8340 tablet_service.cc:1467] Processing CreateTablet for tablet 402de43677c44deab8793da1895d4fb2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9af1e1dd28c144d9b52b73387f91d232]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.665119  8080 tablet_service.cc:1467] Processing CreateTablet for tablet 402de43677c44deab8793da1895d4fb2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9af1e1dd28c144d9b52b73387f91d232]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.665388  8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 402de43677c44deab8793da1895d4fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.665556  8080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 402de43677c44deab8793da1895d4fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.669512  9213 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:02.671101  8341 tablet_service.cc:1467] Processing CreateTablet for tablet 6043da7d2e584af885ff6a5bf8f06912 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9af1e1dd28c144d9b52b73387f91d232]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.671572  8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6043da7d2e584af885ff6a5bf8f06912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.671633  9213 tablet_bootstrap.cc:654] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.677582  8212 tablet_service.cc:1467] Processing CreateTablet for tablet 402de43677c44deab8793da1895d4fb2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9af1e1dd28c144d9b52b73387f91d232]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.678179  8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 402de43677c44deab8793da1895d4fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.679023  8211 tablet_service.cc:1467] Processing CreateTablet for tablet 6043da7d2e584af885ff6a5bf8f06912 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9af1e1dd28c144d9b52b73387f91d232]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.680037  9214 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:02.680168  7949 tablet_service.cc:1467] Processing CreateTablet for tablet 6043da7d2e584af885ff6a5bf8f06912 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9af1e1dd28c144d9b52b73387f91d232]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.680614  7949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6043da7d2e584af885ff6a5bf8f06912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.681882  9214 tablet_bootstrap.cc:654] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.676162  7950 tablet_service.cc:1467] Processing CreateTablet for tablet 007b59de142c4b95966494ba097b1a09 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9af1e1dd28c144d9b52b73387f91d232]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:02.684664  7950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007b59de142c4b95966494ba097b1a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.688784  8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6043da7d2e584af885ff6a5bf8f06912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.691787  9213 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:02.691998  9213 ts_tablet_manager.cc:1367] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.023s	user 0.021s	sys 0.000s
I20240503 02:13:02.692888  9213 raft_consensus.cc:348] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:02.693116  9213 raft_consensus.cc:374] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.693204  9213 raft_consensus.cc:729] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.693439  9213 consensus_queue.cc:260] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:02.694839  9213 ts_tablet_manager.cc:1398] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:13:02.695113  9213 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:02.696751  9213 tablet_bootstrap.cc:654] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.703065  9217 tablet_bootstrap.cc:492] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:02.706012  9213 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:02.706225  9213 ts_tablet_manager.cc:1367] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.007s
I20240503 02:13:02.706624  9217 tablet_bootstrap.cc:654] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.707079  9213 raft_consensus.cc:348] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.707345  9213 raft_consensus.cc:374] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.707427  9213 raft_consensus.cc:729] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.707742  9213 consensus_queue.cc:260] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.708993  9213 ts_tablet_manager.cc:1398] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:13:02.714058  9219 tablet_bootstrap.cc:492] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:02.714753  9217 tablet_bootstrap.cc:492] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:02.714804  9214 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:02.714928  9217 ts_tablet_manager.cc:1367] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.002s
I20240503 02:13:02.714977  9214 ts_tablet_manager.cc:1367] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.035s	user 0.016s	sys 0.016s
I20240503 02:13:02.715613  9217 raft_consensus.cc:348] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.715790  9217 raft_consensus.cc:374] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.715884  9217 raft_consensus.cc:729] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.715796  9214 raft_consensus.cc:348] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.715960  9219 tablet_bootstrap.cc:654] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.716044  9214 raft_consensus.cc:374] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.716145  9214 raft_consensus.cc:729] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.716154  9217 consensus_queue.cc:260] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.716778  9217 ts_tablet_manager.cc:1398] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:13:02.716392  9214 consensus_queue.cc:260] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.717151  9217 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:02.718142  9219 tablet_bootstrap.cc:492] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:02.718305  9219 ts_tablet_manager.cc:1367] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:13:02.718428  9214 ts_tablet_manager.cc:1398] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:13:02.718741  9214 tablet_bootstrap.cc:492] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:02.719197  9217 tablet_bootstrap.cc:654] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.719134  9219 raft_consensus.cc:348] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.719360  9219 raft_consensus.cc:374] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.719467  9219 raft_consensus.cc:729] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.719744  9219 consensus_queue.cc:260] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.720356  9214 tablet_bootstrap.cc:654] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.720425  9219 ts_tablet_manager.cc:1398] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:13:02.720754  9219 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:02.722965  9219 tablet_bootstrap.cc:654] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.730335  9217 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:02.730623  9217 ts_tablet_manager.cc:1367] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.013s	user 0.011s	sys 0.001s
I20240503 02:13:02.731720  9217 raft_consensus.cc:348] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:02.732059  9217 raft_consensus.cc:374] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.732153  9217 raft_consensus.cc:729] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.732483  9217 consensus_queue.cc:260] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:02.733338  9217 ts_tablet_manager.cc:1398] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:13:02.741722  9219 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:02.741904  9219 ts_tablet_manager.cc:1367] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.021s	user 0.007s	sys 0.010s
I20240503 02:13:02.742095  9214 tablet_bootstrap.cc:492] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:02.742269  9214 ts_tablet_manager.cc:1367] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.024s	user 0.004s	sys 0.018s
I20240503 02:13:02.742650  9219 raft_consensus.cc:348] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.742841  9219 raft_consensus.cc:374] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.742914  9219 raft_consensus.cc:729] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.743023  9214 raft_consensus.cc:348] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.743237  9214 raft_consensus.cc:374] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.743146  9219 consensus_queue.cc:260] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.743314  9214 raft_consensus.cc:729] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.743664  9219 ts_tablet_manager.cc:1398] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:13:02.743571  9214 consensus_queue.cc:260] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.743942  9219 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:02.744053  9214 ts_tablet_manager.cc:1398] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:13:02.745519  9219 tablet_bootstrap.cc:654] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.747612  9219 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:02.747769  9219 ts_tablet_manager.cc:1367] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 02:13:02.748618  9219 raft_consensus.cc:348] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:02.748802  9219 raft_consensus.cc:374] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.748878  9219 raft_consensus.cc:729] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.749049  9219 consensus_queue.cc:260] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:02.749418  9219 ts_tablet_manager.cc:1398] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:13:02.838071  8966 raft_consensus.cc:482] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:02.838270  8966 raft_consensus.cc:504] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.838996  8966 leader_election.cc:288] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:02.839596  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6043da7d2e584af885ff6a5bf8f06912" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:13:02.839625  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6043da7d2e584af885ff6a5bf8f06912" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:02.839833  7971 raft_consensus.cc:2454] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:13:02.839881  8361 raft_consensus.cc:2454] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:13:02.840327  8165 leader_election.cc:302] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:02.840850  8966 raft_consensus.cc:2781] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:02.840988  8966 raft_consensus.cc:482] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:02.841099  8966 raft_consensus.cc:3037] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.843189  8966 raft_consensus.cc:504] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.844156  8966 leader_election.cc:288] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Requested vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:02.844347  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6043da7d2e584af885ff6a5bf8f06912" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:02.844405  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6043da7d2e584af885ff6a5bf8f06912" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:02.844597  8361 raft_consensus.cc:3037] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.844615  7971 raft_consensus.cc:3037] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.847601  8361 raft_consensus.cc:2454] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:13:02.847589  7971 raft_consensus.cc:2454] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:13:02.848119  8168 leader_election.cc:302] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64; no voters: 
I20240503 02:13:02.848495  8966 raft_consensus.cc:2781] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:02.848654  8966 raft_consensus.cc:686] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Becoming Leader. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Running, Role: LEADER
I20240503 02:13:02.848948  8966 consensus_queue.cc:237] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.851842  7822 catalog_manager.cc:5484] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3). New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:02.895058  9186 raft_consensus.cc:482] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:02.895287  9186 raft_consensus.cc:504] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:02.895973  9186 leader_election.cc:288] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:02.896536  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007b59de142c4b95966494ba097b1a09" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:02.896585  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007b59de142c4b95966494ba097b1a09" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:13:02.896766  8231 raft_consensus.cc:2454] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:02.896837  7971 raft_consensus.cc:2454] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:02.897266  8035 leader_election.cc:302] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:02.897552  9186 raft_consensus.cc:2781] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:02.897661  9186 raft_consensus.cc:482] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:02.897749  9186 raft_consensus.cc:3037] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.899747  9186 raft_consensus.cc:504] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:02.900292  9186 leader_election.cc:288] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Requested vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:02.900802  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007b59de142c4b95966494ba097b1a09" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:02.901028  8231 raft_consensus.cc:3037] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.901000  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "007b59de142c4b95966494ba097b1a09" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:02.901216  7971 raft_consensus.cc:3037] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.903040  8231 raft_consensus.cc:2454] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:02.903352  7971 raft_consensus.cc:2454] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:02.903501  8035 leader_election.cc:302] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:02.903851  9185 raft_consensus.cc:2781] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:02.904011  9185 raft_consensus.cc:686] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Becoming Leader. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:13:02.904367  9185 consensus_queue.cc:237] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:02.907255  7822 catalog_manager.cc:5484] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2). New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:02.924810  9121 raft_consensus.cc:482] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:02.924983  9121 raft_consensus.cc:504] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.925586  9121 leader_election.cc:288] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:02.926090  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "402de43677c44deab8793da1895d4fb2" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:02.926105  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "402de43677c44deab8793da1895d4fb2" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:02.926317  8231 raft_consensus.cc:2454] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:13:02.926324  8102 raft_consensus.cc:2454] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:13:02.926812  8295 leader_election.cc:302] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64; no voters: 
I20240503 02:13:02.927084  9121 raft_consensus.cc:2781] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:02.927208  9121 raft_consensus.cc:482] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:02.927315  9121 raft_consensus.cc:3037] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.929281  9121 raft_consensus.cc:504] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.929955  9121 leader_election.cc:288] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Requested vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:02.930361  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "402de43677c44deab8793da1895d4fb2" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:02.930431  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "402de43677c44deab8793da1895d4fb2" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:02.930569  8231 raft_consensus.cc:3037] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.930651  8102 raft_consensus.cc:3037] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.932507  8102 raft_consensus.cc:2454] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:02.932574  8231 raft_consensus.cc:2454] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:02.933017  8297 leader_election.cc:302] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:02.933353  9121 raft_consensus.cc:2781] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:02.933503  9121 raft_consensus.cc:686] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Running, Role: LEADER
I20240503 02:13:02.933817  9121 consensus_queue.cc:237] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:02.936836  7822 catalog_manager.cc:5484] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d7c037d50c241b484ee843b06313a64 (127.0.145.4). New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:03.236994  9226 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:03.246265  8966 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:03.356652  9230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:03.357053  9230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:03.364771  9230 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:03.384393  9121 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:03.389366  9121 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240503 02:13:03.421769  9186 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:03.428192  9185 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 265.814 ms
  time per row: 0.132907 ms
I20240503 02:13:03.792999  7822 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50564:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:13:03.802532  7950 tablet_service.cc:1467] Processing CreateTablet for tablet ceae1c7c45a34b24badbbe23cb9e0176 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c7d72eaa895849fcb02235c1b6e8d08f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.802990  8211 tablet_service.cc:1467] Processing CreateTablet for tablet ceae1c7c45a34b24badbbe23cb9e0176 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c7d72eaa895849fcb02235c1b6e8d08f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.803140  7950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceae1c7c45a34b24badbbe23cb9e0176. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.803330  8082 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea56405515a40f785726bd58ec3d26a (DEFAULT_TABLE table=rebalance_test_table_4 [id=c7d72eaa895849fcb02235c1b6e8d08f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.803493  8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceae1c7c45a34b24badbbe23cb9e0176. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.803732  8082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea56405515a40f785726bd58ec3d26a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.804673  8210 tablet_service.cc:1467] Processing CreateTablet for tablet d3d4b912095244de816d3089a31b8649 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c7d72eaa895849fcb02235c1b6e8d08f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.805073  8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d4b912095244de816d3089a31b8649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.807128  8080 tablet_service.cc:1467] Processing CreateTablet for tablet ceae1c7c45a34b24badbbe23cb9e0176 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c7d72eaa895849fcb02235c1b6e8d08f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.807550  8080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceae1c7c45a34b24badbbe23cb9e0176. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.809072  7949 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea56405515a40f785726bd58ec3d26a (DEFAULT_TABLE table=rebalance_test_table_4 [id=c7d72eaa895849fcb02235c1b6e8d08f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.809484  7949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea56405515a40f785726bd58ec3d26a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.811028  7951 tablet_service.cc:1467] Processing CreateTablet for tablet d3d4b912095244de816d3089a31b8649 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c7d72eaa895849fcb02235c1b6e8d08f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.811424  7951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d4b912095244de816d3089a31b8649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.812052  8341 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea56405515a40f785726bd58ec3d26a (DEFAULT_TABLE table=rebalance_test_table_4 [id=c7d72eaa895849fcb02235c1b6e8d08f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.812222  8340 tablet_service.cc:1467] Processing CreateTablet for tablet d3d4b912095244de816d3089a31b8649 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c7d72eaa895849fcb02235c1b6e8d08f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.812500  8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea56405515a40f785726bd58ec3d26a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.812722  8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d4b912095244de816d3089a31b8649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.823908  9297 tablet_bootstrap.cc:492] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:03.825968  9297 tablet_bootstrap.cc:654] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.831030  9300 tablet_bootstrap.cc:492] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:03.831995  9299 tablet_bootstrap.cc:492] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:03.833204  9300 tablet_bootstrap.cc:654] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.833853  9298 tablet_bootstrap.cc:492] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:03.834710  9299 tablet_bootstrap.cc:654] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.839210  9298 tablet_bootstrap.cc:654] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.842206  9297 tablet_bootstrap.cc:492] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:03.842414  9297 ts_tablet_manager.cc:1367] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.019s	user 0.000s	sys 0.016s
I20240503 02:13:03.842598  9299 tablet_bootstrap.cc:492] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:03.842649  9298 tablet_bootstrap.cc:492] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:03.842697  9300 tablet_bootstrap.cc:492] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:03.842798  9298 ts_tablet_manager.cc:1367] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.001s
I20240503 02:13:03.842861  9300 ts_tablet_manager.cc:1367] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.006s
I20240503 02:13:03.843027  9299 ts_tablet_manager.cc:1367] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.011s	user 0.001s	sys 0.009s
I20240503 02:13:03.843446  9297 raft_consensus.cc:348] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:03.843685  9297 raft_consensus.cc:374] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.843781  9297 raft_consensus.cc:729] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.843688  9298 raft_consensus.cc:348] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:03.843672  9300 raft_consensus.cc:348] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:03.843892  9298 raft_consensus.cc:374] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.843899  9300 raft_consensus.cc:374] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.843993  9300 raft_consensus.cc:729] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.844007  9298 raft_consensus.cc:729] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.844055  9297 consensus_queue.cc:260] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:03.844275  9300 consensus_queue.cc:260] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:03.844296  9298 consensus_queue.cc:260] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:03.844372  9299 raft_consensus.cc:348] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.844961  9300 ts_tablet_manager.cc:1398] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:13:03.845048  9298 ts_tablet_manager.cc:1398] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:13:03.845263  9300 tablet_bootstrap.cc:492] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:03.845336  9298 tablet_bootstrap.cc:492] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:03.847074  9298 tablet_bootstrap.cc:654] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.847290  9299 raft_consensus.cc:374] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.847523  9297 ts_tablet_manager.cc:1398] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 02:13:03.847846  9297 tablet_bootstrap.cc:492] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:03.849618  9297 tablet_bootstrap.cc:654] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.853590  9300 tablet_bootstrap.cc:654] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.854941  9299 raft_consensus.cc:729] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.855391  9298 tablet_bootstrap.cc:492] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:03.855580  9298 ts_tablet_manager.cc:1367] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.007s
I20240503 02:13:03.855365  9299 consensus_queue.cc:260] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.856230  9299 ts_tablet_manager.cc:1398] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.000s
I20240503 02:13:03.856603  9298 raft_consensus.cc:348] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.856844  9298 raft_consensus.cc:374] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.857743  9298 raft_consensus.cc:729] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.858089  9298 consensus_queue.cc:260] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.858808  9299 tablet_bootstrap.cc:492] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:03.859766  9298 ts_tablet_manager.cc:1398] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.002s
I20240503 02:13:03.861277  9299 tablet_bootstrap.cc:654] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.863623  9300 tablet_bootstrap.cc:492] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:03.863623  9297 tablet_bootstrap.cc:492] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:03.863807  9297 ts_tablet_manager.cc:1367] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.016s	user 0.010s	sys 0.002s
I20240503 02:13:03.863818  9300 ts_tablet_manager.cc:1367] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.019s	user 0.005s	sys 0.005s
I20240503 02:13:03.864774  9297 raft_consensus.cc:348] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.865029  9297 raft_consensus.cc:374] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.865105  9297 raft_consensus.cc:729] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.865365  9297 consensus_queue.cc:260] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.866034  9297 ts_tablet_manager.cc:1398] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:13:03.866592  9300 raft_consensus.cc:348] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.866847  9300 raft_consensus.cc:374] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.866981  9300 raft_consensus.cc:729] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.867270  9300 consensus_queue.cc:260] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.868309  9299 tablet_bootstrap.cc:492] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:03.868512  9299 ts_tablet_manager.cc:1367] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.007s
I20240503 02:13:03.868542  9300 ts_tablet_manager.cc:1398] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:13:03.868880  9300 tablet_bootstrap.cc:492] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:03.869390  9299 raft_consensus.cc:348] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.869657  9299 raft_consensus.cc:374] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.869736  9299 raft_consensus.cc:729] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.870002  9299 consensus_queue.cc:260] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.870633  9299 ts_tablet_manager.cc:1398] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 02:13:03.871169  9300 tablet_bootstrap.cc:654] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.873471  9300 tablet_bootstrap.cc:492] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:03.873615  9300 ts_tablet_manager.cc:1367] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.001s
I20240503 02:13:03.874414  9300 raft_consensus.cc:348] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.874702  9300 raft_consensus.cc:374] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.874820  9300 raft_consensus.cc:729] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.875074  9300 consensus_queue.cc:260] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.875638  9300 ts_tablet_manager.cc:1398] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:13:03.877346  9147 raft_consensus.cc:482] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:03.877465  9147 raft_consensus.cc:504] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.878214  9147 leader_election.cc:288] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:03.878703  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea56405515a40f785726bd58ec3d26a" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:03.878917  8102 raft_consensus.cc:2454] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 0.
I20240503 02:13:03.879001  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea56405515a40f785726bd58ec3d26a" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:03.879280  8361 raft_consensus.cc:2454] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 0.
I20240503 02:13:03.879357  7906 leader_election.cc:302] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:03.879714  9147 raft_consensus.cc:2781] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:03.879839  9147 raft_consensus.cc:482] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:03.879933  9147 raft_consensus.cc:3037] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.882226  9147 raft_consensus.cc:504] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.883204  9147 leader_election.cc:288] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:03.883306  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea56405515a40f785726bd58ec3d26a" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:03.883366  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea56405515a40f785726bd58ec3d26a" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:03.883508  8102 raft_consensus.cc:3037] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.883545  8362 raft_consensus.cc:3037] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.885442  8362 raft_consensus.cc:2454] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 1.
I20240503 02:13:03.885450  8102 raft_consensus.cc:2454] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 1.
I20240503 02:13:03.885835  7906 leader_election.cc:302] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:03.886132  9147 raft_consensus.cc:2781] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:03.886261  9147 raft_consensus.cc:686] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Becoming Leader. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Running, Role: LEADER
I20240503 02:13:03.886545  9147 consensus_queue.cc:237] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:03.889390  7822 catalog_manager.cc:5484] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: term changed from 0 to 1, leader changed from <none> to d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:03.984189  9233 raft_consensus.cc:482] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:03.984376  9233 raft_consensus.cc:504] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:03.985235  9233 leader_election.cc:288] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:03.985703  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceae1c7c45a34b24badbbe23cb9e0176" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:13:03.985747  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceae1c7c45a34b24badbbe23cb9e0176" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:03.985942  7969 raft_consensus.cc:2454] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:13:03.985980  8102 raft_consensus.cc:2454] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:13:03.986387  8165 leader_election.cc:302] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:03.986685  9233 raft_consensus.cc:2781] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:03.986806  9233 raft_consensus.cc:482] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:03.986896  9233 raft_consensus.cc:3037] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.988867  9233 raft_consensus.cc:504] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:03.989470  9233 leader_election.cc:288] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Requested vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:03.989974  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceae1c7c45a34b24badbbe23cb9e0176" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:03.989988  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceae1c7c45a34b24badbbe23cb9e0176" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:03.990168  7969 raft_consensus.cc:3037] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.990191  8102 raft_consensus.cc:3037] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:03.992128  8102 raft_consensus.cc:2454] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:13:03.992130  7969 raft_consensus.cc:2454] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:13:03.992604  8167 leader_election.cc:302] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:03.992915  9233 raft_consensus.cc:2781] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:03.993084  9233 raft_consensus.cc:686] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Becoming Leader. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Running, Role: LEADER
I20240503 02:13:03.993391  9233 consensus_queue.cc:237] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:03.996219  7822 catalog_manager.cc:5484] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3). New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:04.067595  8980 raft_consensus.cc:482] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:04.067822  8980 raft_consensus.cc:504] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:04.068542  8980 leader_election.cc:288] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:04.069232  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d4b912095244de816d3089a31b8649" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:04.069149  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d4b912095244de816d3089a31b8649" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:13:04.069468  8231 raft_consensus.cc:2454] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:13:04.069542  7969 raft_consensus.cc:2454] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:13:04.069979  8295 leader_election.cc:302] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:04.070262  8980 raft_consensus.cc:2781] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:04.070376  8980 raft_consensus.cc:482] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:04.070498  8980 raft_consensus.cc:3037] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.072957  8980 raft_consensus.cc:504] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:04.073455  8980 leader_election.cc:288] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Requested vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:04.073887  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d4b912095244de816d3089a31b8649" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:04.074005  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3d4b912095244de816d3089a31b8649" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:04.074139  8231 raft_consensus.cc:3037] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.074229  7969 raft_consensus.cc:3037] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.077093  8231 raft_consensus.cc:2454] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:04.077107  7969 raft_consensus.cc:2454] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:04.077608  8295 leader_election.cc:302] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64; no voters: 
I20240503 02:13:04.077958  8980 raft_consensus.cc:2781] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:04.078169  8980 raft_consensus.cc:686] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Running, Role: LEADER
I20240503 02:13:04.078477  8980 consensus_queue.cc:237] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:04.081431  7822 catalog_manager.cc:5484] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d7c037d50c241b484ee843b06313a64 (127.0.145.4). New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:04.371291  9147 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:04.376950  9310 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:04.386096  9233 consensus_queue.cc:1035] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:04.397634  9235 consensus_queue.cc:1035] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:04.491621  9314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:04.492087  9314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:04.499747  9314 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:04.518663  8980 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:04.523239  9121 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'

INSERT report
    rows total: 2000
    time total: 292.209 ms
  time per row: 0.146105 ms
I20240503 02:13:04.947050  7822 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50564:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:13:04.955763  8211 tablet_service.cc:1467] Processing CreateTablet for tablet 7e966caa651e4e42890be5d79f851a1f (DEFAULT_TABLE table=rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.956008  8080 tablet_service.cc:1467] Processing CreateTablet for tablet 7e966caa651e4e42890be5d79f851a1f (DEFAULT_TABLE table=rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.956247  8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e966caa651e4e42890be5d79f851a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.956396  8080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e966caa651e4e42890be5d79f851a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.956727  8341 tablet_service.cc:1467] Processing CreateTablet for tablet 7e966caa651e4e42890be5d79f851a1f (DEFAULT_TABLE table=rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.957172  8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e966caa651e4e42890be5d79f851a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.957988  8212 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2cedba9f9a4e709623fceeffcfd095 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.958381  8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2cedba9f9a4e709623fceeffcfd095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.963488  7949 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2cedba9f9a4e709623fceeffcfd095 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.963420  8210 tablet_service.cc:1467] Processing CreateTablet for tablet 2448e12efbd44a90b1f8bce9519e73e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.963935  7949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2cedba9f9a4e709623fceeffcfd095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.963974  8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2448e12efbd44a90b1f8bce9519e73e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.967834  8082 tablet_service.cc:1467] Processing CreateTablet for tablet 2448e12efbd44a90b1f8bce9519e73e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.967924  8081 tablet_service.cc:1467] Processing CreateTablet for tablet 4f2cedba9f9a4e709623fceeffcfd095 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.968350  8082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2448e12efbd44a90b1f8bce9519e73e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.968794  8081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2cedba9f9a4e709623fceeffcfd095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.969060  8340 tablet_service.cc:1467] Processing CreateTablet for tablet 2448e12efbd44a90b1f8bce9519e73e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:04.969460  8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2448e12efbd44a90b1f8bce9519e73e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.976804  9382 tablet_bootstrap.cc:492] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:04.979892  9381 tablet_bootstrap.cc:492] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:04.981245  9382 tablet_bootstrap.cc:654] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.981914  9381 tablet_bootstrap.cc:654] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.984843  9383 tablet_bootstrap.cc:492] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:04.986817  9385 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:04.987857  9383 tablet_bootstrap.cc:654] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.989341  9381 tablet_bootstrap.cc:492] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:04.989387  9382 tablet_bootstrap.cc:492] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:04.989537  9382 ts_tablet_manager.cc:1367] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.000s
I20240503 02:13:04.989567  9385 tablet_bootstrap.cc:654] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.989707  9381 ts_tablet_manager.cc:1367] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.010s	user 0.009s	sys 0.000s
I20240503 02:13:04.990460  9382 raft_consensus.cc:348] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:04.990726  9382 raft_consensus.cc:374] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.990813  9382 raft_consensus.cc:729] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.990628  9381 raft_consensus.cc:348] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:04.991130  9381 raft_consensus.cc:374] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.991104  9382 consensus_queue.cc:260] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:04.991715  9382 ts_tablet_manager.cc:1398] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:13:04.992055  9382 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:04.991266  9381 raft_consensus.cc:729] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.992715  9381 consensus_queue.cc:260] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:04.993702  9382 tablet_bootstrap.cc:654] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.002673  9381 ts_tablet_manager.cc:1398] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.013s	user 0.000s	sys 0.002s
I20240503 02:13:05.002766  9385 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:05.002985  9385 ts_tablet_manager.cc:1367] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.016s	user 0.008s	sys 0.000s
I20240503 02:13:05.004139  9385 raft_consensus.cc:348] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:05.004439  9385 raft_consensus.cc:374] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.004544  9385 raft_consensus.cc:729] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.004928  9385 consensus_queue.cc:260] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:05.005631  9385 ts_tablet_manager.cc:1398] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:13:05.006486  9381 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:05.008625  9382 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:05.008780  9383 tablet_bootstrap.cc:492] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:05.008816  9382 ts_tablet_manager.cc:1367] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.017s	user 0.006s	sys 0.010s
I20240503 02:13:05.008949  9383 ts_tablet_manager.cc:1367] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.024s	user 0.008s	sys 0.012s
I20240503 02:13:05.009181  9381 tablet_bootstrap.cc:654] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.009699  9382 raft_consensus.cc:348] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:05.009783  9383 raft_consensus.cc:348] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:05.009990  9383 raft_consensus.cc:374] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.010000  9382 raft_consensus.cc:374] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.010078  9382 raft_consensus.cc:729] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.010092  9383 raft_consensus.cc:729] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.010416  9382 consensus_queue.cc:260] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:05.010392  9383 consensus_queue.cc:260] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:05.011098  9383 ts_tablet_manager.cc:1398] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:13:05.011799  9383 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:05.012588  9381 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:05.012773  9381 ts_tablet_manager.cc:1367] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.003s
I20240503 02:13:05.012790  9382 ts_tablet_manager.cc:1398] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:13:05.013221  9382 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:05.013726  9383 tablet_bootstrap.cc:654] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.013628  9381 raft_consensus.cc:348] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:05.013860  9381 raft_consensus.cc:374] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.013947  9381 raft_consensus.cc:729] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.014235  9235 raft_consensus.cc:482] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:05.014259  9381 consensus_queue.cc:260] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:05.015128  9381 ts_tablet_manager.cc:1398] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 02:13:05.015836  9381 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:05.014425  9235 raft_consensus.cc:504] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:05.017280  9235 leader_election.cc:288] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:05.015560  9382 tablet_bootstrap.cc:654] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.018901  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e966caa651e4e42890be5d79f851a1f" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:05.019222  9381 tablet_bootstrap.cc:654] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.019475  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e966caa651e4e42890be5d79f851a1f" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:05.019754  8362 raft_consensus.cc:2454] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:13:05.020375  8168 leader_election.cc:302] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64; no voters: 
I20240503 02:13:05.020761  9235 raft_consensus.cc:2781] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:05.020887  9235 raft_consensus.cc:482] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:05.021023  9235 raft_consensus.cc:3037] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.019186  8102 raft_consensus.cc:2454] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:13:05.023471  9383 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:05.023483  9381 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:05.023619  9381 ts_tablet_manager.cc:1367] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.001s
I20240503 02:13:05.023640  9383 ts_tablet_manager.cc:1367] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.009s
I20240503 02:13:05.024201  9235 raft_consensus.cc:504] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:05.024379  9381 raft_consensus.cc:348] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:05.024557  9381 raft_consensus.cc:374] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.024642  9381 raft_consensus.cc:729] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.024852  9381 consensus_queue.cc:260] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:05.024348  9383 raft_consensus.cc:348] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:05.025290  9383 raft_consensus.cc:374] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.025368  9383 raft_consensus.cc:729] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.025429  9381 ts_tablet_manager.cc:1398] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:13:05.025610  9383 consensus_queue.cc:260] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:05.026162  9383 ts_tablet_manager.cc:1398] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:13:05.026268  9382 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:05.026408  9382 ts_tablet_manager.cc:1367] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.013s	user 0.002s	sys 0.004s
I20240503 02:13:05.027192  9382 raft_consensus.cc:348] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:05.027417  9382 raft_consensus.cc:374] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.027792  9382 raft_consensus.cc:729] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.028342  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e966caa651e4e42890be5d79f851a1f" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:05.028055  9382 consensus_queue.cc:260] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:05.028528  8102 raft_consensus.cc:3037] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.029021  9382 ts_tablet_manager.cc:1398] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 02:13:05.031188  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e966caa651e4e42890be5d79f851a1f" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:05.031348  8102 raft_consensus.cc:2454] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:13:05.031405  8362 raft_consensus.cc:3037] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.032507  8167 leader_election.cc:302] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:05.032922  9235 leader_election.cc:288] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:05.033138  9235 raft_consensus.cc:2781] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:05.033353  9235 raft_consensus.cc:686] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Becoming Leader. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Running, Role: LEADER
I20240503 02:13:05.033594  9235 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:05.034175  8362 raft_consensus.cc:2454] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:13:05.036906  7828 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3). New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:05.125555  9235 raft_consensus.cc:482] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:05.125772  9235 raft_consensus.cc:504] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:05.126550  9235 leader_election.cc:288] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:05.127090  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2448e12efbd44a90b1f8bce9519e73e0" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:05.127100  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2448e12efbd44a90b1f8bce9519e73e0" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:05.127332  8362 raft_consensus.cc:2454] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:13:05.127344  8102 raft_consensus.cc:2454] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:13:05.127782  8167 leader_election.cc:302] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:05.128085  9235 raft_consensus.cc:2781] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:05.128202  9235 raft_consensus.cc:482] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:05.128294  9235 raft_consensus.cc:3037] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.131166  9235 raft_consensus.cc:504] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:05.131850  9235 leader_election.cc:288] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:05.132346  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2448e12efbd44a90b1f8bce9519e73e0" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:05.132470  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2448e12efbd44a90b1f8bce9519e73e0" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:05.132602  8102 raft_consensus.cc:3037] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.132683  8362 raft_consensus.cc:3037] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.135669  8362 raft_consensus.cc:2454] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:13:05.135748  8102 raft_consensus.cc:2454] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:13:05.136221  8168 leader_election.cc:302] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:05.136613  9235 raft_consensus.cc:2781] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:05.136757  9235 raft_consensus.cc:686] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Becoming Leader. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Running, Role: LEADER
I20240503 02:13:05.137074  9235 consensus_queue.cc:237] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:05.140131  7828 catalog_manager.cc:5484] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3). New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:05.211421  9368 raft_consensus.cc:482] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:05.211598  9368 raft_consensus.cc:504] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:05.212280  9368 leader_election.cc:288] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:05.212838  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2cedba9f9a4e709623fceeffcfd095" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:05.213093  8102 raft_consensus.cc:2454] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 0.
I20240503 02:13:05.213534  7906 leader_election.cc:302] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:05.213804  9368 raft_consensus.cc:2781] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:05.213939  9368 raft_consensus.cc:482] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:05.214037  9368 raft_consensus.cc:3037] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.216189  9368 raft_consensus.cc:504] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:05.216850  9368 leader_election.cc:288] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:05.217306  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2cedba9f9a4e709623fceeffcfd095" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:05.217505  8102 raft_consensus.cc:3037] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.218796  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2cedba9f9a4e709623fceeffcfd095" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:05.218924  8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f2cedba9f9a4e709623fceeffcfd095" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:05.219094  8232 raft_consensus.cc:2454] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 0.
I20240503 02:13:05.219875  8102 raft_consensus.cc:2454] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 1.
I20240503 02:13:05.220237  7906 leader_election.cc:302] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d986fcfb926e47baa7d2d5b99f697334; no voters: 2942efc0ecb448d2ba2f6bca20fbb29c
I20240503 02:13:05.220571  9368 raft_consensus.cc:2781] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:05.220712  9368 raft_consensus.cc:686] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Becoming Leader. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Running, Role: LEADER
I20240503 02:13:05.220980  9368 consensus_queue.cc:237] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:05.224086  7828 catalog_manager.cc:5484] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: term changed from 0 to 1, leader changed from <none> to d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:05.426379  9235 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.433739  9233 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.630687  9368 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:05.630488  9398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:05.631001  9398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:05.640444  9398 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:05.650380  8230 raft_consensus.cc:3037] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.654376  9410 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.678112  9233 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240503 02:13:05.683836  9235 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 243.731 ms
  time per row: 0.121866 ms
I20240503 02:13:06.057047  7828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50564:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:13:06.067276  8341 tablet_service.cc:1467] Processing CreateTablet for tablet c45c5a14b85447c78fe2a27be4392c88 (DEFAULT_TABLE table=rebalance_test_table_6 [id=938b875abcf9426ead26522c081d6b7c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.067751  8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45c5a14b85447c78fe2a27be4392c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.067819  8080 tablet_service.cc:1467] Processing CreateTablet for tablet c45c5a14b85447c78fe2a27be4392c88 (DEFAULT_TABLE table=rebalance_test_table_6 [id=938b875abcf9426ead26522c081d6b7c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.068266  8080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45c5a14b85447c78fe2a27be4392c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.068420  8340 tablet_service.cc:1467] Processing CreateTablet for tablet f231fe5dfd9e4c87b90bad75818a7060 (DEFAULT_TABLE table=rebalance_test_table_6 [id=938b875abcf9426ead26522c081d6b7c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.068830  8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f231fe5dfd9e4c87b90bad75818a7060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.069221  7949 tablet_service.cc:1467] Processing CreateTablet for tablet c45c5a14b85447c78fe2a27be4392c88 (DEFAULT_TABLE table=rebalance_test_table_6 [id=938b875abcf9426ead26522c081d6b7c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.069613  7949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45c5a14b85447c78fe2a27be4392c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.075112  7950 tablet_service.cc:1467] Processing CreateTablet for tablet f231fe5dfd9e4c87b90bad75818a7060 (DEFAULT_TABLE table=rebalance_test_table_6 [id=938b875abcf9426ead26522c081d6b7c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.075567  7950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f231fe5dfd9e4c87b90bad75818a7060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.079515  8081 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 (DEFAULT_TABLE table=rebalance_test_table_6 [id=938b875abcf9426ead26522c081d6b7c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.079972  8081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.079941  8212 tablet_service.cc:1467] Processing CreateTablet for tablet f231fe5dfd9e4c87b90bad75818a7060 (DEFAULT_TABLE table=rebalance_test_table_6 [id=938b875abcf9426ead26522c081d6b7c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.080341  8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f231fe5dfd9e4c87b90bad75818a7060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.081070  9467 tablet_bootstrap.cc:492] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:06.087679  8340 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 (DEFAULT_TABLE table=rebalance_test_table_6 [id=938b875abcf9426ead26522c081d6b7c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.088294  8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.088497  9468 tablet_bootstrap.cc:492] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:06.088871  7951 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 (DEFAULT_TABLE table=rebalance_test_table_6 [id=938b875abcf9426ead26522c081d6b7c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.089341  7951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.089357  9467 tablet_bootstrap.cc:654] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.091557  9468 tablet_bootstrap.cc:654] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.092613  9467 tablet_bootstrap.cc:492] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:06.092798  9467 ts_tablet_manager.cc:1367] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.000s
I20240503 02:13:06.093703  9467 raft_consensus.cc:348] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:06.093951  9467 raft_consensus.cc:374] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.094098  9467 raft_consensus.cc:729] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.094463  9467 consensus_queue.cc:260] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:06.095336  9467 ts_tablet_manager.cc:1398] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:13:06.095737  9467 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:06.098044  9467 tablet_bootstrap.cc:654] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.103652  9473 tablet_bootstrap.cc:492] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:06.106024  9472 tablet_bootstrap.cc:492] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:06.106163  9473 tablet_bootstrap.cc:654] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.108378  9472 tablet_bootstrap.cc:654] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.128353  9472 tablet_bootstrap.cc:492] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:06.128785  9472 ts_tablet_manager.cc:1367] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.023s	user 0.004s	sys 0.000s
I20240503 02:13:06.130012  9472 raft_consensus.cc:348] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:06.130410  9472 raft_consensus.cc:374] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.130578  9472 raft_consensus.cc:729] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.130980  9472 consensus_queue.cc:260] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:06.132402  9472 ts_tablet_manager.cc:1398] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:13:06.132728  9472 tablet_bootstrap.cc:492] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:06.133445  9467 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:06.133783  9467 ts_tablet_manager.cc:1367] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.038s	user 0.017s	sys 0.020s
I20240503 02:13:06.134416  9472 tablet_bootstrap.cc:654] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.134893  9467 raft_consensus.cc:348] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:06.135363  9467 raft_consensus.cc:374] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.135516  9467 raft_consensus.cc:729] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.135917  9467 consensus_queue.cc:260] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:06.137765  9467 ts_tablet_manager.cc:1398] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:13:06.137926  9468 tablet_bootstrap.cc:492] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:06.138127  9468 ts_tablet_manager.cc:1367] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.050s	user 0.014s	sys 0.023s
I20240503 02:13:06.139401  9468 raft_consensus.cc:348] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:06.139716  9468 raft_consensus.cc:374] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.139853  9468 raft_consensus.cc:729] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.140183  9468 consensus_queue.cc:260] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:06.140761  9468 ts_tablet_manager.cc:1398] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 02:13:06.141057  9468 tablet_bootstrap.cc:492] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:06.142632  9468 tablet_bootstrap.cc:654] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.144688  9468 tablet_bootstrap.cc:492] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:06.144840  9468 ts_tablet_manager.cc:1367] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 02:13:06.145053  9473 tablet_bootstrap.cc:492] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:06.145208  9473 ts_tablet_manager.cc:1367] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.042s	user 0.014s	sys 0.011s
I20240503 02:13:06.145536  9468 raft_consensus.cc:348] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:06.145726  9468 raft_consensus.cc:374] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.145787  9468 raft_consensus.cc:729] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.146057  9473 raft_consensus.cc:348] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:06.146273  9473 raft_consensus.cc:374] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.146365  9473 raft_consensus.cc:729] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.146521  9472 tablet_bootstrap.cc:492] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:06.146713  9472 ts_tablet_manager.cc:1367] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.000s
I20240503 02:13:06.146870  9473 consensus_queue.cc:260] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:06.147541  9473 ts_tablet_manager.cc:1398] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:13:06.147454  9472 raft_consensus.cc:348] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:06.147661  9472 raft_consensus.cc:374] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.147725  9472 raft_consensus.cc:729] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.148205  9468 consensus_queue.cc:260] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:06.148741  9468 ts_tablet_manager.cc:1398] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240503 02:13:06.149067  9468 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:06.150638  9468 tablet_bootstrap.cc:654] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.151635  9472 consensus_queue.cc:260] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:06.152958  9472 ts_tablet_manager.cc:1398] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240503 02:13:06.153256  9472 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:06.154904  9472 tablet_bootstrap.cc:654] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.159251  9468 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:06.159418  9468 ts_tablet_manager.cc:1367] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.010s	user 0.001s	sys 0.003s
I20240503 02:13:06.160184  9468 raft_consensus.cc:348] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:06.160408  9468 raft_consensus.cc:374] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.160490  9468 raft_consensus.cc:729] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.160706  9468 consensus_queue.cc:260] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:06.161242  9468 ts_tablet_manager.cc:1398] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:13:06.162117  9472 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:06.162273  9472 ts_tablet_manager.cc:1367] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.000s
I20240503 02:13:06.163060  9472 raft_consensus.cc:348] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:06.163287  9472 raft_consensus.cc:374] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.163378  9472 raft_consensus.cc:729] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.163591  9472 consensus_queue.cc:260] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:06.164227  9472 ts_tablet_manager.cc:1398] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:13:06.168040  9368 raft_consensus.cc:482] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:06.168229  9368 raft_consensus.cc:504] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:06.168828  9368 leader_election.cc:288] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:06.169245  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45c5a14b85447c78fe2a27be4392c88" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:06.169310  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45c5a14b85447c78fe2a27be4392c88" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:06.169471  8362 raft_consensus.cc:2454] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 0.
I20240503 02:13:06.169556  8102 raft_consensus.cc:2454] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 0.
I20240503 02:13:06.169901  7907 leader_election.cc:302] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:06.170213  9368 raft_consensus.cc:2781] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:06.170336  9368 raft_consensus.cc:482] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:06.170441  9368 raft_consensus.cc:3037] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.172384  9368 raft_consensus.cc:504] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:06.172960  9368 leader_election.cc:288] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 election: Requested vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:06.173420  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45c5a14b85447c78fe2a27be4392c88" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:06.173440  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45c5a14b85447c78fe2a27be4392c88" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:06.173632  8102 raft_consensus.cc:3037] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.173630  8362 raft_consensus.cc:3037] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.175913  8102 raft_consensus.cc:2454] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 1.
I20240503 02:13:06.175911  8362 raft_consensus.cc:2454] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 1.
I20240503 02:13:06.176331  7906 leader_election.cc:302] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:06.176761  9368 raft_consensus.cc:2781] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:06.176920  9368 raft_consensus.cc:686] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Becoming Leader. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Running, Role: LEADER
I20240503 02:13:06.177210  9368 consensus_queue.cc:237] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:06.180213  7826 catalog_manager.cc:5484] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: term changed from 0 to 1, leader changed from <none> to d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:06.252580  8980 raft_consensus.cc:482] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:06.252795  8980 raft_consensus.cc:504] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:06.253429  8980 leader_election.cc:288] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:06.253985  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:06.254034  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:13:06.254249  7969 raft_consensus.cc:2454] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:13:06.254233  8102 raft_consensus.cc:2454] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:13:06.254694  8297 leader_election.cc:302] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:06.254976  8980 raft_consensus.cc:2781] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:06.255077  8980 raft_consensus.cc:482] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:06.255203  8980 raft_consensus.cc:3037] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.257273  8980 raft_consensus.cc:504] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:06.257882  8980 leader_election.cc:288] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:06.258324  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:06.258541  7969 raft_consensus.cc:3037] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.260486  7969 raft_consensus.cc:2454] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:06.260924  8295 leader_election.cc:302] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:06.260903  8102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:06.261138  8102 raft_consensus.cc:3037] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.261214  8980 raft_consensus.cc:2781] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:06.261649  8980 raft_consensus.cc:686] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Running, Role: LEADER
I20240503 02:13:06.261956  8980 consensus_queue.cc:237] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:06.263417  8102 raft_consensus.cc:2454] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:06.265069  7826 catalog_manager.cc:5484] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d7c037d50c241b484ee843b06313a64 (127.0.145.4). New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:06.299247  9368 raft_consensus.cc:482] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:06.299474  9368 raft_consensus.cc:504] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:06.300026  9368 leader_election.cc:288] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:06.300586  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f231fe5dfd9e4c87b90bad75818a7060" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:06.300647  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f231fe5dfd9e4c87b90bad75818a7060" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:06.300834  8232 raft_consensus.cc:2454] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 0.
I20240503 02:13:06.300896  8362 raft_consensus.cc:2454] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 0.
I20240503 02:13:06.301395  7904 leader_election.cc:302] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:06.301748  9368 raft_consensus.cc:2781] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:06.301832  9368 raft_consensus.cc:482] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:06.301935  9368 raft_consensus.cc:3037] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.304304  9368 raft_consensus.cc:504] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:06.305083  9368 leader_election.cc:288] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 election: Requested vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:06.305554  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f231fe5dfd9e4c87b90bad75818a7060" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:06.305609  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f231fe5dfd9e4c87b90bad75818a7060" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:06.305799  8362 raft_consensus.cc:3037] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.305811  8232 raft_consensus.cc:3037] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.308723  8362 raft_consensus.cc:2454] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 1.
I20240503 02:13:06.308706  8232 raft_consensus.cc:2454] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 1.
I20240503 02:13:06.309268  7904 leader_election.cc:302] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:06.309645  9368 raft_consensus.cc:2781] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:06.309811  9368 raft_consensus.cc:686] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Becoming Leader. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Running, Role: LEADER
I20240503 02:13:06.310082  9368 consensus_queue.cc:237] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:06.313334  7826 catalog_manager.cc:5484] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: term changed from 0 to 1, leader changed from <none> to d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:06.650164  8980 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.657184  9481 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.701236  9368 consensus_queue.cc:1035] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.707334  9410 consensus_queue.cc:1035] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:06.756410  9484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:06.756814  9484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:06.764703  9484 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:06.766134  9410 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.771203  9410 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'

INSERT report
    rows total: 2000
    time total: 219.139 ms
  time per row: 0.10957 ms
I20240503 02:13:07.137876  7828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50564:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:13:07.148077  7950 tablet_service.cc:1467] Processing CreateTablet for tablet ecb5c78a22a74728ab40f1da8f90388c (DEFAULT_TABLE table=rebalance_test_table_7 [id=635c8841bff54466a804d93ade171b2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.148586  7950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb5c78a22a74728ab40f1da8f90388c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.148504  8212 tablet_service.cc:1467] Processing CreateTablet for tablet ecb5c78a22a74728ab40f1da8f90388c (DEFAULT_TABLE table=rebalance_test_table_7 [id=635c8841bff54466a804d93ade171b2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.148913  8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb5c78a22a74728ab40f1da8f90388c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.149799  8341 tablet_service.cc:1467] Processing CreateTablet for tablet de9c2a3396864696af88be1ca12ee0f1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=635c8841bff54466a804d93ade171b2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.150053  8211 tablet_service.cc:1467] Processing CreateTablet for tablet de9c2a3396864696af88be1ca12ee0f1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=635c8841bff54466a804d93ade171b2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.150213  8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de9c2a3396864696af88be1ca12ee0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.150429  8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de9c2a3396864696af88be1ca12ee0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.152029  8210 tablet_service.cc:1467] Processing CreateTablet for tablet c23b06764ce54acb8388de94aea4cd4b (DEFAULT_TABLE table=rebalance_test_table_7 [id=635c8841bff54466a804d93ade171b2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.152463  8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23b06764ce54acb8388de94aea4cd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.155414  8340 tablet_service.cc:1467] Processing CreateTablet for tablet c23b06764ce54acb8388de94aea4cd4b (DEFAULT_TABLE table=rebalance_test_table_7 [id=635c8841bff54466a804d93ade171b2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.155869  8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23b06764ce54acb8388de94aea4cd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.158444  7951 tablet_service.cc:1467] Processing CreateTablet for tablet c23b06764ce54acb8388de94aea4cd4b (DEFAULT_TABLE table=rebalance_test_table_7 [id=635c8841bff54466a804d93ade171b2f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.158936  7951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23b06764ce54acb8388de94aea4cd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.159062  8081 tablet_service.cc:1467] Processing CreateTablet for tablet ecb5c78a22a74728ab40f1da8f90388c (DEFAULT_TABLE table=rebalance_test_table_7 [id=635c8841bff54466a804d93ade171b2f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.159468  8081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb5c78a22a74728ab40f1da8f90388c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.164029  8080 tablet_service.cc:1467] Processing CreateTablet for tablet de9c2a3396864696af88be1ca12ee0f1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=635c8841bff54466a804d93ade171b2f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:07.164501  8080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de9c2a3396864696af88be1ca12ee0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.165985  9548 tablet_bootstrap.cc:492] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:07.168264  9548 tablet_bootstrap.cc:654] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.173411  9550 tablet_bootstrap.cc:492] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:07.175736  9550 tablet_bootstrap.cc:654] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.177343  9552 tablet_bootstrap.cc:492] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:07.177374  9549 tablet_bootstrap.cc:492] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:07.179747  9549 tablet_bootstrap.cc:654] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.181211  9552 tablet_bootstrap.cc:654] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.184293  9549 tablet_bootstrap.cc:492] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:07.185053  9549 ts_tablet_manager.cc:1367] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.002s
I20240503 02:13:07.186407  9548 tablet_bootstrap.cc:492] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:07.186584  9548 ts_tablet_manager.cc:1367] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.021s	user 0.011s	sys 0.007s
I20240503 02:13:07.186930  9552 tablet_bootstrap.cc:492] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:07.186980  9550 tablet_bootstrap.cc:492] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:07.187150  9552 ts_tablet_manager.cc:1367] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.004s
I20240503 02:13:07.187206  9550 ts_tablet_manager.cc:1367] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.014s	user 0.001s	sys 0.006s
I20240503 02:13:07.187601  9548 raft_consensus.cc:348] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:07.187914  9548 raft_consensus.cc:374] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.188071  9548 raft_consensus.cc:729] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.187983  9552 raft_consensus.cc:348] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:07.188202  9552 raft_consensus.cc:374] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.188117  9550 raft_consensus.cc:348] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:07.188154  9549 raft_consensus.cc:348] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:07.188330  9552 raft_consensus.cc:729] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.188349  9549 raft_consensus.cc:374] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.188436  9549 raft_consensus.cc:729] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.188424  9550 raft_consensus.cc:374] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.188532  9550 raft_consensus.cc:729] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.188488  9548 consensus_queue.cc:260] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:07.188760  9549 consensus_queue.cc:260] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:07.188704  9552 consensus_queue.cc:260] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:07.188866  9550 consensus_queue.cc:260] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:07.189417  9549 ts_tablet_manager.cc:1398] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240503 02:13:07.189406  9550 ts_tablet_manager.cc:1398] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:13:07.189716  9550 tablet_bootstrap.cc:492] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:07.190037  9548 ts_tablet_manager.cc:1398] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 02:13:07.190315  9548 tablet_bootstrap.cc:492] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:07.191610  9550 tablet_bootstrap.cc:654] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.191913  9549 tablet_bootstrap.cc:492] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:07.191990  9548 tablet_bootstrap.cc:654] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.194752  9549 tablet_bootstrap.cc:654] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.203963  9549 tablet_bootstrap.cc:492] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:07.204232  9549 ts_tablet_manager.cc:1367] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.005s
I20240503 02:13:07.204666  9552 ts_tablet_manager.cc:1398] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.017s	user 0.002s	sys 0.000s
I20240503 02:13:07.205289  9552 tablet_bootstrap.cc:492] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:07.205189  9549 raft_consensus.cc:348] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:07.205516  9549 raft_consensus.cc:374] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.205624  9549 raft_consensus.cc:729] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.205926  9549 consensus_queue.cc:260] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:07.206537  9549 ts_tablet_manager.cc:1398] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:13:07.206883  9549 tablet_bootstrap.cc:492] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:07.207139  9552 tablet_bootstrap.cc:654] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.208932  9549 tablet_bootstrap.cc:654] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.211817  9548 tablet_bootstrap.cc:492] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:07.212088  9548 ts_tablet_manager.cc:1367] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.022s	user 0.011s	sys 0.007s
I20240503 02:13:07.212963  9550 tablet_bootstrap.cc:492] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:07.213208  9550 ts_tablet_manager.cc:1367] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.024s	user 0.007s	sys 0.008s
I20240503 02:13:07.212947  9548 raft_consensus.cc:348] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:07.213881  9548 raft_consensus.cc:374] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.214010  9548 raft_consensus.cc:729] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.214239  9550 raft_consensus.cc:348] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:07.214401  9548 consensus_queue.cc:260] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:07.214695  9550 raft_consensus.cc:374] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.214833  9550 raft_consensus.cc:729] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.215269  9550 consensus_queue.cc:260] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:07.216044  9552 tablet_bootstrap.cc:492] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:07.216044  9548 ts_tablet_manager.cc:1398] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:13:07.216223  9552 ts_tablet_manager.cc:1367] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.001s
I20240503 02:13:07.216851  9550 ts_tablet_manager.cc:1398] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:13:07.217043  9552 raft_consensus.cc:348] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:07.217289  9552 raft_consensus.cc:374] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.217373  9552 raft_consensus.cc:729] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.217880  9552 consensus_queue.cc:260] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:07.220002  9552 ts_tablet_manager.cc:1398] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.001s
I20240503 02:13:07.221494  9549 tablet_bootstrap.cc:492] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:07.221676  9549 ts_tablet_manager.cc:1367] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.003s
I20240503 02:13:07.222532  9549 raft_consensus.cc:348] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:07.222781  9549 raft_consensus.cc:374] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.222913  9549 raft_consensus.cc:729] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.223167  9549 consensus_queue.cc:260] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:07.223662  9549 ts_tablet_manager.cc:1398] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:13:07.272615  9481 raft_consensus.cc:482] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:07.272866  9481 raft_consensus.cc:504] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:07.273592  9481 leader_election.cc:288] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:07.274187  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23b06764ce54acb8388de94aea4cd4b" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:07.274260  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23b06764ce54acb8388de94aea4cd4b" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:13:07.274449  8232 raft_consensus.cc:2454] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:13:07.274514  7969 raft_consensus.cc:2454] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:13:07.274957  8295 leader_election.cc:302] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:07.275250  9481 raft_consensus.cc:2781] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:07.275354  9481 raft_consensus.cc:482] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:07.275468  9481 raft_consensus.cc:3037] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.276125  9491 raft_consensus.cc:482] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:07.276293  9491 raft_consensus.cc:504] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:07.277057  9491 leader_election.cc:288] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:07.277488  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb5c78a22a74728ab40f1da8f90388c" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:13:07.277570  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb5c78a22a74728ab40f1da8f90388c" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:07.277704  7969 raft_consensus.cc:2454] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:13:07.277827  8101 raft_consensus.cc:2454] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 0.
I20240503 02:13:07.278154  8165 leader_election.cc:302] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:07.278118  9481 raft_consensus.cc:504] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:07.278713  9491 raft_consensus.cc:2781] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:07.278839  9491 raft_consensus.cc:482] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:07.278841  9481 leader_election.cc:288] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Requested vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:07.278980  9491 raft_consensus.cc:3037] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.279358  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23b06764ce54acb8388de94aea4cd4b" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:07.279574  7969 raft_consensus.cc:3037] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.280292  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23b06764ce54acb8388de94aea4cd4b" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:07.280521  8232 raft_consensus.cc:3037] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.282028  9491 raft_consensus.cc:504] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:07.282554  7969 raft_consensus.cc:2454] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:07.282967  9491 leader_election.cc:288] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Requested vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:07.283057  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb5c78a22a74728ab40f1da8f90388c" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:07.283267  7971 raft_consensus.cc:3037] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.283221  8295 leader_election.cc:302] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:07.283485  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb5c78a22a74728ab40f1da8f90388c" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:07.283555  9481 raft_consensus.cc:2781] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:07.283641  8101 raft_consensus.cc:3037] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.283715  9481 raft_consensus.cc:686] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Running, Role: LEADER
I20240503 02:13:07.284263  9481 consensus_queue.cc:237] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:07.285784  8101 raft_consensus.cc:2454] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:13:07.286226  8167 leader_election.cc:302] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:07.286375  7971 raft_consensus.cc:2454] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 1.
I20240503 02:13:07.286597  9491 raft_consensus.cc:2781] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:07.286867  8232 raft_consensus.cc:2454] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:07.287618  9491 raft_consensus.cc:686] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Becoming Leader. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Running, Role: LEADER
I20240503 02:13:07.287909  9491 consensus_queue.cc:237] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:07.290135  7828 catalog_manager.cc:5484] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d7c037d50c241b484ee843b06313a64 (127.0.145.4). New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:07.290899  7824 catalog_manager.cc:5484] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3). New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:07.291836  9487 raft_consensus.cc:482] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:07.291991  9487 raft_consensus.cc:504] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:07.292726  9487 leader_election.cc:288] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:07.293407  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de9c2a3396864696af88be1ca12ee0f1" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:07.293658  8232 raft_consensus.cc:2454] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:13:07.293742  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de9c2a3396864696af88be1ca12ee0f1" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:07.293974  8101 raft_consensus.cc:2454] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:13:07.294070  8295 leader_election.cc:302] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64; no voters: 
I20240503 02:13:07.294422  9487 raft_consensus.cc:2781] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:07.294574  9487 raft_consensus.cc:482] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:07.294709  9487 raft_consensus.cc:3037] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.296555  9487 raft_consensus.cc:504] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:07.297065  9487 leader_election.cc:288] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:07.297489  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de9c2a3396864696af88be1ca12ee0f1" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:07.297729  8101 raft_consensus.cc:3037] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.297747  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de9c2a3396864696af88be1ca12ee0f1" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:07.297956  8232 raft_consensus.cc:3037] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.299729  8101 raft_consensus.cc:2454] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:07.299826  8232 raft_consensus.cc:2454] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:07.300161  8297 leader_election.cc:302] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:07.300484  9487 raft_consensus.cc:2781] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:07.300628  9487 raft_consensus.cc:686] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Running, Role: LEADER
I20240503 02:13:07.300879  9487 consensus_queue.cc:237] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } }
I20240503 02:13:07.304147  7824 catalog_manager.cc:5484] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d7c037d50c241b484ee843b06313a64 (127.0.145.4). New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } }
W20240503 02:13:07.709997  9566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:07.710405  9566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:07.718430  9566 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 02:13:07.792655  9487 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.797700  9481 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.806083  9487 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.853392  9487 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.859767  9491 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.877460  9491 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 293.207 ms
  time per row: 0.146603 ms
I20240503 02:13:08.189891  7825 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50564:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:13:08.199388  7950 tablet_service.cc:1467] Processing CreateTablet for tablet 4af5884849ec4ac3891ebf0f08ee2cff (DEFAULT_TABLE table=rebalance_test_table_8 [id=8da59f26450d4d25a86bc61be2943c37]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.199503  8212 tablet_service.cc:1467] Processing CreateTablet for tablet 4af5884849ec4ac3891ebf0f08ee2cff (DEFAULT_TABLE table=rebalance_test_table_8 [id=8da59f26450d4d25a86bc61be2943c37]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.199990  8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af5884849ec4ac3891ebf0f08ee2cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.199987  7950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af5884849ec4ac3891ebf0f08ee2cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.200433  7951 tablet_service.cc:1467] Processing CreateTablet for tablet 72e2bb75da654e8ab45bba8ac077f28c (DEFAULT_TABLE table=rebalance_test_table_8 [id=8da59f26450d4d25a86bc61be2943c37]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.200804  7951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e2bb75da654e8ab45bba8ac077f28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.202706  7949 tablet_service.cc:1467] Processing CreateTablet for tablet 834aaba965c8443c862e98a070e36cac (DEFAULT_TABLE table=rebalance_test_table_8 [id=8da59f26450d4d25a86bc61be2943c37]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.203125  7949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 834aaba965c8443c862e98a070e36cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.203395  8340 tablet_service.cc:1467] Processing CreateTablet for tablet 72e2bb75da654e8ab45bba8ac077f28c (DEFAULT_TABLE table=rebalance_test_table_8 [id=8da59f26450d4d25a86bc61be2943c37]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.203809  8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e2bb75da654e8ab45bba8ac077f28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.207370  8080 tablet_service.cc:1467] Processing CreateTablet for tablet 4af5884849ec4ac3891ebf0f08ee2cff (DEFAULT_TABLE table=rebalance_test_table_8 [id=8da59f26450d4d25a86bc61be2943c37]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.207767  8080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af5884849ec4ac3891ebf0f08ee2cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.207954  8341 tablet_service.cc:1467] Processing CreateTablet for tablet 834aaba965c8443c862e98a070e36cac (DEFAULT_TABLE table=rebalance_test_table_8 [id=8da59f26450d4d25a86bc61be2943c37]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.208591  8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 834aaba965c8443c862e98a070e36cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.211109  8081 tablet_service.cc:1467] Processing CreateTablet for tablet 72e2bb75da654e8ab45bba8ac077f28c (DEFAULT_TABLE table=rebalance_test_table_8 [id=8da59f26450d4d25a86bc61be2943c37]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.211526  8081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e2bb75da654e8ab45bba8ac077f28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.214780  8211 tablet_service.cc:1467] Processing CreateTablet for tablet 834aaba965c8443c862e98a070e36cac (DEFAULT_TABLE table=rebalance_test_table_8 [id=8da59f26450d4d25a86bc61be2943c37]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.215273  8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 834aaba965c8443c862e98a070e36cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.224020  9626 tablet_bootstrap.cc:492] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:08.226863  9626 tablet_bootstrap.cc:654] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.230945  9625 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:08.230823  9627 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:08.232897  9625 tablet_bootstrap.cc:654] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.233512  9627 tablet_bootstrap.cc:654] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.233523  9628 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:08.236402  9628 tablet_bootstrap.cc:654] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.239099  9627 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:08.239099  9626 tablet_bootstrap.cc:492] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:08.239271  9626 ts_tablet_manager.cc:1367] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.015s	user 0.004s	sys 0.009s
I20240503 02:13:08.239271  9627 ts_tablet_manager.cc:1367] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.001s
I20240503 02:13:08.239409  9625 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:08.239701  9625 ts_tablet_manager.cc:1367] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.003s
I20240503 02:13:08.240100  9627 raft_consensus.cc:348] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.240084  9626 raft_consensus.cc:348] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.240320  9626 raft_consensus.cc:374] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.240316  9627 raft_consensus.cc:374] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.240412  9627 raft_consensus.cc:729] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.240412  9626 raft_consensus.cc:729] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.240644  9627 consensus_queue.cc:260] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.240640  9626 consensus_queue.cc:260] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.241190  9626 ts_tablet_manager.cc:1398] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:13:08.241209  9627 ts_tablet_manager.cc:1398] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:13:08.241477  9627 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:08.241477  9626 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:08.243189  9627 tablet_bootstrap.cc:654] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.243032  9625 raft_consensus.cc:348] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:08.243499  9625 raft_consensus.cc:374] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.243650  9625 raft_consensus.cc:729] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.244037  9625 consensus_queue.cc:260] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:08.246131  9625 ts_tablet_manager.cc:1398] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240503 02:13:08.246529  9625 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:08.246871  9628 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:08.247093  9628 ts_tablet_manager.cc:1367] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.015s	user 0.011s	sys 0.000s
I20240503 02:13:08.248070  9628 raft_consensus.cc:348] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:08.248426  9628 raft_consensus.cc:374] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.248574  9628 raft_consensus.cc:729] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.248939  9628 consensus_queue.cc:260] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:08.249518  9625 tablet_bootstrap.cc:654] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.249716  9628 ts_tablet_manager.cc:1398] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:13:08.251147  9628 tablet_bootstrap.cc:492] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:08.253589  9628 tablet_bootstrap.cc:654] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.255687  9627 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:08.255904  9627 ts_tablet_manager.cc:1367] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.014s	user 0.009s	sys 0.002s
I20240503 02:13:08.255939  9626 tablet_bootstrap.cc:654] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.257115  9627 raft_consensus.cc:348] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:08.257761  9627 raft_consensus.cc:374] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.257882  9627 raft_consensus.cc:729] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.258639  9627 consensus_queue.cc:260] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:08.261468  9627 ts_tablet_manager.cc:1398] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:13:08.261895  9628 tablet_bootstrap.cc:492] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:08.262143  9628 ts_tablet_manager.cc:1367] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.011s	user 0.009s	sys 0.000s
I20240503 02:13:08.262202  9625 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:08.262431  9625 ts_tablet_manager.cc:1367] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.000s
I20240503 02:13:08.263216  9626 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:08.263164  9628 raft_consensus.cc:348] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.263415  9628 raft_consensus.cc:374] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.263512  9628 raft_consensus.cc:729] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.263630  9626 ts_tablet_manager.cc:1367] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.022s	user 0.006s	sys 0.001s
I20240503 02:13:08.263759  9628 consensus_queue.cc:260] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.264398  9628 ts_tablet_manager.cc:1398] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:13:08.264775  9625 raft_consensus.cc:348] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.265043  9625 raft_consensus.cc:374] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.265133  9625 raft_consensus.cc:729] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.265105  9626 raft_consensus.cc:348] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.265408  9626 raft_consensus.cc:374] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.265554  9626 raft_consensus.cc:729] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.265460  9625 consensus_queue.cc:260] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.265970  9626 consensus_queue.cc:260] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.266633  9626 ts_tablet_manager.cc:1398] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240503 02:13:08.267258  9625 ts_tablet_manager.cc:1398] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 02:13:08.267627  9625 tablet_bootstrap.cc:492] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:08.269632  9625 tablet_bootstrap.cc:654] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.269954  9481 raft_consensus.cc:482] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.270133  9481 raft_consensus.cc:504] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.270901  9481 leader_election.cc:288] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:08.271993  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "834aaba965c8443c862e98a070e36cac" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:08.272212  8231 raft_consensus.cc:2454] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 0.
I20240503 02:13:08.272869  8295 leader_election.cc:302] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64; no voters: 
I20240503 02:13:08.273037  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "834aaba965c8443c862e98a070e36cac" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:13:08.274027  9481 raft_consensus.cc:2781] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:08.274147  9481 raft_consensus.cc:482] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240503 02:13:08.274138  8295 leader_election.cc:341] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:08.274286  9481 raft_consensus.cc:3037] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.276772  9481 raft_consensus.cc:504] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.277402  9481 leader_election.cc:288] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Requested vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:08.278239  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "834aaba965c8443c862e98a070e36cac" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:08.278576  8231 raft_consensus.cc:3037] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.279188  9625 tablet_bootstrap.cc:492] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:08.279621  9625 ts_tablet_manager.cc:1367] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.012s	user 0.010s	sys 0.000s
I20240503 02:13:08.279572  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "834aaba965c8443c862e98a070e36cac" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:08.280586  9625 raft_consensus.cc:348] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
W20240503 02:13:08.280727  8295 leader_election.cc:341] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:08.280870  9625 raft_consensus.cc:374] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.281004  9625 raft_consensus.cc:729] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.281303  9625 consensus_queue.cc:260] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.281711  8231 raft_consensus.cc:2454] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 1.
I20240503 02:13:08.281971  9625 ts_tablet_manager.cc:1398] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:13:08.282130  8295 leader_election.cc:302] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64; no voters: d986fcfb926e47baa7d2d5b99f697334
I20240503 02:13:08.282636  9481 raft_consensus.cc:2781] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:08.282848  9481 raft_consensus.cc:686] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Becoming Leader. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Running, Role: LEADER
I20240503 02:13:08.283109  9481 consensus_queue.cc:237] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.286237  7822 catalog_manager.cc:5484] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d7c037d50c241b484ee843b06313a64 (127.0.145.4). New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:08.355113  9185 raft_consensus.cc:482] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.355329  9185 raft_consensus.cc:504] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:08.356052  9185 leader_election.cc:288] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:08.356472  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af5884849ec4ac3891ebf0f08ee2cff" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:13:08.356559  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af5884849ec4ac3891ebf0f08ee2cff" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:08.356704  7971 raft_consensus.cc:2454] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:08.356824  8231 raft_consensus.cc:2454] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:08.357126  8035 leader_election.cc:302] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:08.357388  9185 raft_consensus.cc:2781] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:08.357496  9185 raft_consensus.cc:482] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:08.357589  9185 raft_consensus.cc:3037] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.359753  9185 raft_consensus.cc:504] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:08.360306  9185 leader_election.cc:288] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Requested vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:08.360855  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af5884849ec4ac3891ebf0f08ee2cff" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:08.360867  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af5884849ec4ac3891ebf0f08ee2cff" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:08.361088  7971 raft_consensus.cc:3037] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.361101  8231 raft_consensus.cc:3037] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.363430  8231 raft_consensus.cc:2454] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:08.363454  7971 raft_consensus.cc:2454] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:08.363833  8035 leader_election.cc:302] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:08.364166  9185 raft_consensus.cc:2781] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:08.364637  9185 raft_consensus.cc:686] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Becoming Leader. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:13:08.364970  9185 consensus_queue.cc:237] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:08.368062  7828 catalog_manager.cc:5484] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2). New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:08.489110  9185 raft_consensus.cc:482] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.489336  9185 raft_consensus.cc:504] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.490020  9185 leader_election.cc:288] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:08.490540  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e2bb75da654e8ab45bba8ac077f28c" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:08.490650  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e2bb75da654e8ab45bba8ac077f28c" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:13:08.490828  8362 raft_consensus.cc:2454] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:08.490906  7971 raft_consensus.cc:2454] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:08.491315  8038 leader_election.cc:302] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:08.491634  9185 raft_consensus.cc:2781] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:08.491717  9185 raft_consensus.cc:482] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:08.491823  9185 raft_consensus.cc:3037] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.493768  9185 raft_consensus.cc:504] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.494410  9185 leader_election.cc:288] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Requested vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:08.494905  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e2bb75da654e8ab45bba8ac077f28c" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:08.495146  8362 raft_consensus.cc:3037] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.495131  7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e2bb75da654e8ab45bba8ac077f28c" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:08.495359  7971 raft_consensus.cc:3037] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.497160  8362 raft_consensus.cc:2454] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:08.497577  8038 leader_election.cc:302] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:08.497591  7971 raft_consensus.cc:2454] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:08.498050  9185 raft_consensus.cc:2781] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:08.498208  9185 raft_consensus.cc:686] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Becoming Leader. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:13:08.498471  9185 consensus_queue.cc:237] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.499253  9410 raft_consensus.cc:482] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.499408  9410 raft_consensus.cc:504] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } }
I20240503 02:13:08.500211  9410 leader_election.cc:288] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:08.500680  8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "834aaba965c8443c862e98a070e36cac" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:08.500869  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "834aaba965c8443c862e98a070e36cac" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:08.501013  8231 raft_consensus.cc:2379] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d986fcfb926e47baa7d2d5b99f697334 in current term 1: Already voted for candidate 8d7c037d50c241b484ee843b06313a64 in this term.
I20240503 02:13:08.501672  7907 leader_election.cc:302] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d986fcfb926e47baa7d2d5b99f697334; no voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64
I20240503 02:13:08.501984  9410 raft_consensus.cc:3037] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.501771  7828 catalog_manager.cc:5484] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2). New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:08.504127  9410 raft_consensus.cc:2726] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:08.786754  9481 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.795146  9481 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.836412  9185 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.844135  9639 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:08.924204  9642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:08.924554  9642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:08.932555  9642 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 02:13:08.996019  9652 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.005641  9652 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 294.186 ms
  time per row: 0.147093 ms
I20240503 02:13:09.374658  7829 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50564:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:13:09.384678  8341 tablet_service.cc:1467] Processing CreateTablet for tablet b64f66ace86e42f3ac213ba85bcab584 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5f6139f12f9b4ab18ce35a8831b39150]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.384678  8081 tablet_service.cc:1467] Processing CreateTablet for tablet b64f66ace86e42f3ac213ba85bcab584 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5f6139f12f9b4ab18ce35a8831b39150]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.385159  8081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64f66ace86e42f3ac213ba85bcab584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.385152  8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64f66ace86e42f3ac213ba85bcab584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.385423  8080 tablet_service.cc:1467] Processing CreateTablet for tablet aeaee03abc644807b8278cfb78e4f02b (DEFAULT_TABLE table=rebalance_test_table_9 [id=5f6139f12f9b4ab18ce35a8831b39150]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.385825  8080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeaee03abc644807b8278cfb78e4f02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.387941  8082 tablet_service.cc:1467] Processing CreateTablet for tablet 45fe6ec58eed40f1898d4a2f5727cfa1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5f6139f12f9b4ab18ce35a8831b39150]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.388362  8082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45fe6ec58eed40f1898d4a2f5727cfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.388942  8212 tablet_service.cc:1467] Processing CreateTablet for tablet b64f66ace86e42f3ac213ba85bcab584 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5f6139f12f9b4ab18ce35a8831b39150]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.389039  8211 tablet_service.cc:1467] Processing CreateTablet for tablet aeaee03abc644807b8278cfb78e4f02b (DEFAULT_TABLE table=rebalance_test_table_9 [id=5f6139f12f9b4ab18ce35a8831b39150]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.389391  8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64f66ace86e42f3ac213ba85bcab584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.389874  8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeaee03abc644807b8278cfb78e4f02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.396277  7950 tablet_service.cc:1467] Processing CreateTablet for tablet 45fe6ec58eed40f1898d4a2f5727cfa1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5f6139f12f9b4ab18ce35a8831b39150]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.396296  8340 tablet_service.cc:1467] Processing CreateTablet for tablet aeaee03abc644807b8278cfb78e4f02b (DEFAULT_TABLE table=rebalance_test_table_9 [id=5f6139f12f9b4ab18ce35a8831b39150]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.396879  7950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45fe6ec58eed40f1898d4a2f5727cfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.396886  8340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeaee03abc644807b8278cfb78e4f02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.399379  9707 tablet_bootstrap.cc:492] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:09.401371  9707 tablet_bootstrap.cc:654] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.406737  8210 tablet_service.cc:1467] Processing CreateTablet for tablet 45fe6ec58eed40f1898d4a2f5727cfa1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5f6139f12f9b4ab18ce35a8831b39150]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.407457  8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45fe6ec58eed40f1898d4a2f5727cfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.431888  9709 tablet_bootstrap.cc:492] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:09.432626  9710 tablet_bootstrap.cc:492] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:09.433756  9707 tablet_bootstrap.cc:492] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:09.434015  9707 ts_tablet_manager.cc:1367] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.035s	user 0.020s	sys 0.012s
I20240503 02:13:09.435161  9707 raft_consensus.cc:348] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:09.435595  9707 raft_consensus.cc:374] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.435745  9707 raft_consensus.cc:729] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.436072  9707 consensus_queue.cc:260] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:09.436895  9707 ts_tablet_manager.cc:1398] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:13:09.437258  9707 tablet_bootstrap.cc:492] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:09.437907  9710 tablet_bootstrap.cc:654] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.439178  9709 tablet_bootstrap.cc:654] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.440676  9707 tablet_bootstrap.cc:654] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.442342  9710 tablet_bootstrap.cc:492] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334: No bootstrap required, opened a new log
I20240503 02:13:09.442596  9710 ts_tablet_manager.cc:1367] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.002s
I20240503 02:13:09.443876  9710 raft_consensus.cc:348] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:09.444269  9710 raft_consensus.cc:374] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.444898  9706 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:09.445881  9707 tablet_bootstrap.cc:492] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:09.446089  9707 ts_tablet_manager.cc:1367] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I20240503 02:13:09.446739  9710 raft_consensus.cc:729] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.446849  9706 tablet_bootstrap.cc:654] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.446987  9707 raft_consensus.cc:348] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.447036  9710 consensus_queue.cc:260] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:09.447237  9707 raft_consensus.cc:374] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.447326  9707 raft_consensus.cc:729] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.448107  9710 ts_tablet_manager.cc:1398] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240503 02:13:09.448318  9709 tablet_bootstrap.cc:492] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:09.448509  9709 ts_tablet_manager.cc:1367] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.000s
I20240503 02:13:09.447618  9707 consensus_queue.cc:260] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.449779  9707 ts_tablet_manager.cc:1398] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 02:13:09.450218  9707 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:09.449290  9709 raft_consensus.cc:348] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.451480  9709 raft_consensus.cc:374] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.451633  9709 raft_consensus.cc:729] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.451988  9709 consensus_queue.cc:260] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.454041  9707 tablet_bootstrap.cc:654] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.455451  9709 ts_tablet_manager.cc:1398] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240503 02:13:09.455467  9706 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:09.455660  9706 ts_tablet_manager.cc:1367] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.011s	user 0.000s	sys 0.009s
I20240503 02:13:09.455873  9709 tablet_bootstrap.cc:492] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:09.456907  9706 raft_consensus.cc:348] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.457306  9706 raft_consensus.cc:374] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.457480  9706 raft_consensus.cc:729] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.457938  9706 consensus_queue.cc:260] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.459096  9706 ts_tablet_manager.cc:1398] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 02:13:09.459478  9706 tablet_bootstrap.cc:492] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:09.460075  9709 tablet_bootstrap.cc:654] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.461587  9706 tablet_bootstrap.cc:654] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.465967  9707 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: No bootstrap required, opened a new log
I20240503 02:13:09.466164  9707 ts_tablet_manager.cc:1367] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.004s
I20240503 02:13:09.466842  9706 tablet_bootstrap.cc:492] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64: No bootstrap required, opened a new log
I20240503 02:13:09.467113  9706 ts_tablet_manager.cc:1367] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.006s
I20240503 02:13:09.467224  9707 raft_consensus.cc:348] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.467461  9707 raft_consensus.cc:374] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.467558  9707 raft_consensus.cc:729] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.467810  9707 consensus_queue.cc:260] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.467958  9706 raft_consensus.cc:348] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.468180  9706 raft_consensus.cc:374] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.468273  9706 raft_consensus.cc:729] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.468454  9707 ts_tablet_manager.cc:1398] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:13:09.468549  9706 consensus_queue.cc:260] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.469117  9706 ts_tablet_manager.cc:1398] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:13:09.470445  9709 tablet_bootstrap.cc:492] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:09.470682  9709 ts_tablet_manager.cc:1367] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.015s	user 0.007s	sys 0.000s
I20240503 02:13:09.471506  9709 raft_consensus.cc:348] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:09.471735  9709 raft_consensus.cc:374] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.471830  9709 raft_consensus.cc:729] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.472113  9709 consensus_queue.cc:260] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:09.472824  9709 ts_tablet_manager.cc:1398] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:13:09.474263  9709 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:09.476058  9709 tablet_bootstrap.cc:654] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.478037  9709 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c: No bootstrap required, opened a new log
I20240503 02:13:09.478179  9709 ts_tablet_manager.cc:1367] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.001s
I20240503 02:13:09.478870  9709 raft_consensus.cc:348] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.479038  9709 raft_consensus.cc:374] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.479142  9709 raft_consensus.cc:729] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.479349  9709 consensus_queue.cc:260] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.479856  9709 ts_tablet_manager.cc:1398] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 02:13:09.525091  9185 raft_consensus.cc:482] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:09.525285  9185 raft_consensus.cc:504] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.525944  9185 leader_election.cc:288] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:09.526387  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:09.526458  8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:09.526638  8362 raft_consensus.cc:2454] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:09.526718  8230 raft_consensus.cc:2454] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:09.527091  8038 leader_election.cc:302] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:09.527357  9185 raft_consensus.cc:2781] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:09.527468  9185 raft_consensus.cc:482] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:09.527585  9185 raft_consensus.cc:3037] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.529549  9185 raft_consensus.cc:504] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.530186  9185 leader_election.cc:288] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Requested vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:09.530757  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:09.530797  8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:09.531013  8230 raft_consensus.cc:3037] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.531013  8362 raft_consensus.cc:3037] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.533210  8230 raft_consensus.cc:2454] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:09.533218  8362 raft_consensus.cc:2454] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:09.533656  8038 leader_election.cc:302] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:09.533970  9185 raft_consensus.cc:2781] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:09.534153  9185 raft_consensus.cc:686] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Becoming Leader. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:13:09.534407  9185 consensus_queue.cc:237] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.537689  7829 catalog_manager.cc:5484] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2). New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:09.574692  9185 raft_consensus.cc:482] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:09.574944  9185 raft_consensus.cc:504] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:09.575737  9185 leader_election.cc:288] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:09.576112  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45fe6ec58eed40f1898d4a2f5727cfa1" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334" is_pre_election: true
I20240503 02:13:09.576227  8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45fe6ec58eed40f1898d4a2f5727cfa1" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:09.576356  7969 raft_consensus.cc:2454] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:09.576488  8230 raft_consensus.cc:2454] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:09.576875  8035 leader_election.cc:302] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:09.577172  9185 raft_consensus.cc:2781] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:09.577275  9185 raft_consensus.cc:482] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:09.577371  9185 raft_consensus.cc:3037] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.579375  9185 raft_consensus.cc:504] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:09.579939  9185 leader_election.cc:288] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Requested vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:09.580430  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45fe6ec58eed40f1898d4a2f5727cfa1" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:09.580515  8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45fe6ec58eed40f1898d4a2f5727cfa1" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:09.580617  7969 raft_consensus.cc:3037] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.580737  8230 raft_consensus.cc:3037] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.582993  7969 raft_consensus.cc:2454] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:09.583309  8230 raft_consensus.cc:2454] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:09.583483  8035 leader_election.cc:302] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:09.583861  9185 raft_consensus.cc:2781] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:09.584018  9185 raft_consensus.cc:686] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Becoming Leader. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:13:09.584324  9185 consensus_queue.cc:237] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } }
I20240503 02:13:09.587703  7829 catalog_manager.cc:5484] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2). New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:09.608160  9185 raft_consensus.cc:482] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:09.608399  9185 raft_consensus.cc:504] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.609220  9185 leader_election.cc:288] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:09.609767  8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64f66ace86e42f3ac213ba85bcab584" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:13:09.610018  8230 raft_consensus.cc:2454] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:09.610481  8035 leader_election.cc:302] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:09.610914  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64f66ace86e42f3ac213ba85bcab584" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:09.611116  8362 raft_consensus.cc:2454] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 0.
I20240503 02:13:09.611397  9185 raft_consensus.cc:2781] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:09.611502  9185 raft_consensus.cc:482] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:09.611611  9185 raft_consensus.cc:3037] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.614632  9185 raft_consensus.cc:504] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.615340  9185 leader_election.cc:288] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Requested vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:09.615828  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64f66ace86e42f3ac213ba85bcab584" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:09.616076  8362 raft_consensus.cc:3037] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.616359  8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64f66ace86e42f3ac213ba85bcab584" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:09.616559  8230 raft_consensus.cc:3037] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.618624  8230 raft_consensus.cc:2454] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:09.618976  8362 raft_consensus.cc:2454] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 1.
I20240503 02:13:09.619155  8035 leader_election.cc:302] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:09.619511  9185 raft_consensus.cc:2781] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:09.619673  9185 raft_consensus.cc:686] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Becoming Leader. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:13:09.619967  9185 consensus_queue.cc:237] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } }
I20240503 02:13:09.623368  7829 catalog_manager.cc:5484] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2). New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:10.030069  9722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:10.030459  9722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:10.038641  9722 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:10.052726  9677 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.073729  9185 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.077558  9185 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.088712  9185 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240503 02:13:10.099916  9185 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.114750  9677 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:10.389698  8167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.2:35309, user_credentials={real_user=slave}} blocked reactor thread for 49720.2us

INSERT report
    rows total: 2000
    time total: 270.949 ms
  time per row: 0.135474 ms
I20240503 02:13:10.517163   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.5:44891
--local_ip_for_outbound_sockets=127.0.145.5
--tserver_master_addrs=127.0.145.62:36865
--webserver_port=35987
--webserver_interface=127.0.145.5
--builtin_ntp_servers=127.0.145.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:13:10.894913  9781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:10.895231  9781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:10.895349  9781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:10.903156  9781 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:10.903307  9781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:10.903484  9781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.5
I20240503 02:13:10.914321  9781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.5:44891
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.145.5
--webserver_port=35987
--tserver_master_addrs=127.0.145.62:36865
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.5
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:10.916018  9781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:10.918049  9781 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:10.932891  9788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:10.933687  9787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:10.934384  9790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:10.935123  9781 server_base.cc:1034] running on GCE node
I20240503 02:13:10.935808  9781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:10.937263  9781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:10.938534  9781 hybrid_clock.cc:648] HybridClock initialized: now 1714702390938471 us; error 59 us; skew 500 ppm
I20240503 02:13:10.938942  9781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:10.947436  9781 webserver.cc:460] Webserver started at http://127.0.145.5:35987/ using document root <none> and password file <none>
I20240503 02:13:10.948246  9781 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:10.948364  9781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:10.954010  9781 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240503 02:13:10.957190  9796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:13:10.958855  9781 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:13:10.959038  9781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "03f99420df704238ab300ad6e58d878c"
format_stamp: "Formatted at 2024-05-03 02:12:53 on dist-test-slave-gq9h"
I20240503 02:13:10.959887  9781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:10.975625  9781 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:13:10.976483  9781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:10.976779  9781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:10.977761  9781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:10.979354  9781 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:10.979442  9781 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:13:10.979576  9781 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:10.979631  9781 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:13:11.012356  9781 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.5:44891
I20240503 02:13:11.012394  9908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.5:44891 every 8 connection(s)
I20240503 02:13:11.013957  9781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:13:11.019605   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 9781
I20240503 02:13:11.020251   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.6:40397
--local_ip_for_outbound_sockets=127.0.145.6
--tserver_master_addrs=127.0.145.62:36865
--webserver_port=45689
--webserver_interface=127.0.145.6
--builtin_ntp_servers=127.0.145.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:13:11.028054  9909 heartbeater.cc:347] Connected to a master server at 127.0.145.62:36865
I20240503 02:13:11.028463  9909 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:11.029599  9909 heartbeater.cc:511] Master 127.0.145.62:36865 requested a full tablet report, sending...
I20240503 02:13:11.031495  7825 ts_manager.cc:194] Re-registered known tserver with Master: 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891)
I20240503 02:13:11.032825  7825 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.5:45129
W20240503 02:13:11.389050  9913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:11.389470  9913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:11.389636  9913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:11.398329  9913 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:11.398458  9913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:11.398581  9913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.6
I20240503 02:13:11.408171  9913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.6:40397
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.145.6
--webserver_port=45689
--tserver_master_addrs=127.0.145.62:36865
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.6
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:11.409478  9913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:11.411286  9913 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:11.424830  9919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:11.425673  9920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:11.426540  9922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:11.427435  9913 server_base.cc:1034] running on GCE node
I20240503 02:13:11.428057  9913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:11.429014  9913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:11.430217  9913 hybrid_clock.cc:648] HybridClock initialized: now 1714702391430164 us; error 46 us; skew 500 ppm
I20240503 02:13:11.430542  9913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:11.432847  9913 webserver.cc:460] Webserver started at http://127.0.145.6:45689/ using document root <none> and password file <none>
I20240503 02:13:11.433609  9913 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:11.433724  9913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:11.439630  9913 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240503 02:13:11.442804  9928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:13:11.444284  9913 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240503 02:13:11.444485  9913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "b85e85e631ab4e04a86bfe5a9f0aa512"
format_stamp: "Formatted at 2024-05-03 02:12:53 on dist-test-slave-gq9h"
I20240503 02:13:11.445524  9913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:11.462567  9913 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:13:11.463421  9913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:11.463702  9913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:11.464697  9913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:11.466240  9913 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:11.466317  9913 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:13:11.466439  9913 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:11.466496  9913 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:13:11.508987  9913 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.6:40397
I20240503 02:13:11.509027 10040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.6:40397 every 8 connection(s)
I20240503 02:13:11.510572  9913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:13:11.513406   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 9913
I20240503 02:13:11.514079   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.7:39415
--local_ip_for_outbound_sockets=127.0.145.7
--tserver_master_addrs=127.0.145.62:36865
--webserver_port=45823
--webserver_interface=127.0.145.7
--builtin_ntp_servers=127.0.145.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:13:11.526721 10041 heartbeater.cc:347] Connected to a master server at 127.0.145.62:36865
I20240503 02:13:11.527115 10041 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:11.528020 10041 heartbeater.cc:511] Master 127.0.145.62:36865 requested a full tablet report, sending...
I20240503 02:13:11.529618  7829 ts_manager.cc:194] Re-registered known tserver with Master: b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397)
I20240503 02:13:11.530808  7829 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.6:51171
W20240503 02:13:11.907138 10044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:11.907467 10044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:11.907582 10044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:11.915829 10044 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:11.915992 10044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:11.916177 10044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.7
I20240503 02:13:11.926751 10044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.7:39415
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.145.7
--webserver_port=45823
--tserver_master_addrs=127.0.145.62:36865
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.7
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:11.928294 10044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:11.930459 10044 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:11.948714 10051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:11.949265 10054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:11.949353 10052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:11.951304 10044 server_base.cc:1034] running on GCE node
I20240503 02:13:11.952153 10044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:11.953428 10044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:11.954722 10044 hybrid_clock.cc:648] HybridClock initialized: now 1714702391954641 us; error 54 us; skew 500 ppm
I20240503 02:13:11.955214 10044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:11.958469 10044 webserver.cc:460] Webserver started at http://127.0.145.7:45823/ using document root <none> and password file <none>
I20240503 02:13:11.959602 10044 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:11.959785 10044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:11.967902 10044 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:13:11.972343 10060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:13:11.974349 10044 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240503 02:13:11.974640 10044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d826d4dae2aa4e05b2f64c9b17a1126f"
format_stamp: "Formatted at 2024-05-03 02:12:54 on dist-test-slave-gq9h"
I20240503 02:13:11.976050 10044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:12.000000 10044 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:13:12.000911 10044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:12.001185 10044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:12.002161 10044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:12.003809 10044 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:12.003887 10044 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:13:12.003999 10044 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:12.004050 10044 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:13:12.036399  9909 heartbeater.cc:503] Master 127.0.145.62:36865 was elected leader, sending a full tablet report...
I20240503 02:13:12.043556 10044 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.7:39415
I20240503 02:13:12.043596 10172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.7:39415 every 8 connection(s)
I20240503 02:13:12.045003 10044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:13:12.049625   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 10044
I20240503 02:13:12.050428   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.8:38885
--local_ip_for_outbound_sockets=127.0.145.8
--tserver_master_addrs=127.0.145.62:36865
--webserver_port=46757
--webserver_interface=127.0.145.8
--builtin_ntp_servers=127.0.145.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:13:12.060700 10173 heartbeater.cc:347] Connected to a master server at 127.0.145.62:36865
I20240503 02:13:12.061069 10173 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:12.062175 10173 heartbeater.cc:511] Master 127.0.145.62:36865 requested a full tablet report, sending...
I20240503 02:13:12.063742  7829 ts_manager.cc:194] Re-registered known tserver with Master: d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
I20240503 02:13:12.064916  7829 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.7:34605
W20240503 02:13:12.417299 10177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:12.417608 10177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:13:12.417747 10177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:12.425571 10177 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:13:12.425701 10177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:13:12.425864 10177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.8
I20240503 02:13:12.435741 10177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.8:38885
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.145.8
--webserver_port=46757
--tserver_master_addrs=127.0.145.62:36865
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.8
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:13:12.437245 10177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:13:12.439123 10177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:13:12.453059 10183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:13:12.453859 10184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:12.454525 10177 server_base.cc:1034] running on GCE node
W20240503 02:13:12.455528 10186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:13:12.456194 10177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:13:12.457268 10177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:13:12.458470 10177 hybrid_clock.cc:648] HybridClock initialized: now 1714702392458419 us; error 45 us; skew 500 ppm
I20240503 02:13:12.458890 10177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:13:12.461567 10177 webserver.cc:460] Webserver started at http://127.0.145.8:46757/ using document root <none> and password file <none>
I20240503 02:13:12.462479 10177 fs_manager.cc:359] Metadata directory not provided
I20240503 02:13:12.462635 10177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:13:12.468602 10177 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240503 02:13:12.471652 10193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:13:12.473001 10177 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240503 02:13:12.473165 10177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4"
format_stamp: "Formatted at 2024-05-03 02:12:54 on dist-test-slave-gq9h"
I20240503 02:13:12.473999 10177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:13:12.497668 10177 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:13:12.498520 10177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:13:12.498833 10177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:13:12.499761 10177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:13:12.501349 10177 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:13:12.501420 10177 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:13:12.501504 10177 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:13:12.501546 10177 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:13:12.533593 10041 heartbeater.cc:503] Master 127.0.145.62:36865 was elected leader, sending a full tablet report...
I20240503 02:13:12.534416 10177 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.8:38885
I20240503 02:13:12.534613 10305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.8:38885 every 8 connection(s)
I20240503 02:13:12.536190 10177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:13:12.545521   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 10177
I20240503 02:13:12.555716 10306 heartbeater.cc:347] Connected to a master server at 127.0.145.62:36865
I20240503 02:13:12.556089 10306 heartbeater.cc:464] Registering TS with master...
I20240503 02:13:12.557185 10306 heartbeater.cc:511] Master 127.0.145.62:36865 requested a full tablet report, sending...
I20240503 02:13:12.558890  7829 ts_manager.cc:194] Re-registered known tserver with Master: ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
I20240503 02:13:12.559933  7829 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.8:49869
I20240503 02:13:13.068120 10173 heartbeater.cc:503] Master 127.0.145.62:36865 was elected leader, sending a full tablet report...
I20240503 02:13:13.562963 10306 heartbeater.cc:503] Master 127.0.145.62:36865 was elected leader, sending a full tablet report...
I20240503 02:13:13.732071  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:13.749082  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:13.776042  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:13.867789  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:13.890084  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:13.900682 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:13.925640 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:13.976557  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:14.023885  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.024411  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:14.077544  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:14.135357  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.141115 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.171466  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:14.173110  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:14.225436 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.760488  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:14.782593  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:14.800273  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:14.815171 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.816947  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.853614 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.896049  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.898572  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.917577  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:14.919134  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:14.919307  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:14.951424  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.952450 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.980850  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:14.987818 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:15.045107  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:15.374431  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:15.384721  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:15.390954  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:15.418555  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:15.431355  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:15.475373 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:15.489738 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:15.561733  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:15.594220  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:15.675225  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:15.714680  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:15.719018  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:15.724905 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:15.727999  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:15.746628  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:15.754040  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:15.762881 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:15.772574  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:15.787902  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:15.801036  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:15.803068  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:15.840646 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:15.867955 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:15.885668  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:16.416872  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:16.431329  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:16.464764 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:16.469811  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:16.497095 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:16.514526  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:16.518515  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:16.560573  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:16.609788  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:16.631762  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:16.655434  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:16.661377  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:16.684964  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:16.705382 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:16.706460 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:16.720453  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:17.220814  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:17.256325  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:17.280767  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:17.283555  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:17.283540  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:17.362891  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:17.363423  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:17.382030 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:17.394387  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:17.400713  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:17.421228  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:17.425405  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:17.453912 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:17.484047 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:17.532359 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:17.547917  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:17.622952  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:17.637837  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:17.640616  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:17.647449 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:17.676503 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:17.682909  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:17.683357  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:17.729368  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:13:17.757097  8035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.3:38203, user_credentials={real_user=slave}} blocked reactor thread for 70653.8us
I20240503 02:13:17.925099  8101 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:17.973678  7971 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:17.974763 10941 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:17.991644  8361 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:17.992822 10941 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.002403 10942 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:18.003513  8361 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:18.011441  7971 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:18.014135  7827 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
W20240503 02:13:18.059028 10966 log.cc:924] Time spent T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20240503 02:13:18.092772  8101 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:18.096966  7971 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:18.097707 10319 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:18.103456  8361 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:13:18.109767  8038 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f44b2b53de84a87a8757c0d00bfed37. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:18.111074 10319 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:18.110343 10989 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:18.114550  8360 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:18.122058  8362 consensus_queue.cc:237] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:18.131579  7971 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
W20240503 02:13:18.133193  8037 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f44b2b53de84a87a8757c0d00bfed37. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:18.145695  7971 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.149049 10592 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.158341  8101 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.165939 10995 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.172325  7822 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 4 to 5, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
W20240503 02:13:18.182929  8298 consensus_peers.cc:485] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80b4e6aa20f64473be8bcbf47baf66e4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:18.188189 10939 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:18.189077  7971 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:18.199831  7823 catalog_manager.cc:5484] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:18.231022  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:18.242831  8102 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:18.303236  8232 consensus_queue.cc:237] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:18.312039  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:18.320171  8362 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.321171 10996 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.328354  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:18.359167 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:18.359360  7971 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.360329 10192 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.376957  7969 consensus_queue.cc:237] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:18.378576 10996 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:18.385957  7971 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:18.390442  8101 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.391275 10956 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.398144  8231 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.400067  7829 catalog_manager.cc:5484] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:18.404525 10659 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.412993 10659 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:18.419551  8101 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:18.427616  7823 catalog_manager.cc:5484] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
W20240503 02:13:18.429275  7904 consensus_peers.cc:485] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f2cedba9f9a4e709623fceeffcfd095. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:18.436455  8360 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:18.438690  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:18.457789  8232 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:18.479705  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:18.484663  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:13:18.490576  8165 consensus_peers.cc:485] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f14607425da84bb4b162a5789daa844c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:18.513813  8101 consensus_queue.cc:237] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:18.560317  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:18.564652 11070 ts_tablet_manager.cc:902] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
W20240503 02:13:18.567580  8037 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b64f66ace86e42f3ac213ba85bcab584. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:18.567875  8362 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.568583 10989 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.571842  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:18.577147  8232 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.578580 10989 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.581667 11061 ts_tablet_manager.cc:902] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:18.592375 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:18.596899 10942 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:18.602478  8361 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:18.609140  7825 catalog_manager.cc:5484] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:18.644008  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:18.645880  7969 consensus_queue.cc:237] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:18.652640 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:18.654137  8231 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:18.673799  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:18.684700  8362 raft_consensus.cc:1261] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.685788  8101 raft_consensus.cc:1261] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.687390 11083 ts_tablet_manager.cc:902] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:18.687479 10956 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240503 02:13:18.688311 10659 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.695432  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:13:18.701591  7907 consensus_peers.cc:485] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ea56405515a40f785726bd58ec3d26a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:18.702665 11070 tablet_copy_client.cc:308] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:18.714030 11061 tablet_copy_client.cc:308] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:18.717128 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:18.725116 10659 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:18.730686  8101 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:18.732870 11083 tablet_copy_client.cc:308] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:18.734185  7823 catalog_manager.cc:5484] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:18.737396  8362 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:18.744529  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:18.772073  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet 80b4e6aa20f64473be8bcbf47baf66e4 from peer b85e85e631ab4e04a86bfe5a9f0aa512 ({username='slave'} at 127.0.145.6:49885)
I20240503 02:13:18.772398  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet 80b4e6aa20f64473be8bcbf47baf66e4 from peer b85e85e631ab4e04a86bfe5a9f0aa512 at {username='slave'} at 127.0.145.6:49885: session id = b85e85e631ab4e04a86bfe5a9f0aa512-80b4e6aa20f64473be8bcbf47baf66e4
I20240503 02:13:18.776783  8382 tablet_copy_source_session.cc:205] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:18.779234  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 0f44b2b53de84a87a8757c0d00bfed37 from peer ca05c9a8bee148dab8a2a982b8fa2ea4 ({username='slave'} at 127.0.145.8:52015)
I20240503 02:13:18.779551  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 0f44b2b53de84a87a8757c0d00bfed37 from peer ca05c9a8bee148dab8a2a982b8fa2ea4 at {username='slave'} at 127.0.145.8:52015: session id = ca05c9a8bee148dab8a2a982b8fa2ea4-0f44b2b53de84a87a8757c0d00bfed37
I20240503 02:13:18.782177  8121 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 0f44b2b53de84a87a8757c0d00bfed37 from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:48749)
I20240503 02:13:18.782431  8121 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 0f44b2b53de84a87a8757c0d00bfed37 from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:48749: session id = 03f99420df704238ab300ad6e58d878c-0f44b2b53de84a87a8757c0d00bfed37
I20240503 02:13:18.783931 11083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b4e6aa20f64473be8bcbf47baf66e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.785967  8121 tablet_copy_source_session.cc:205] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:18.789757  8122 tablet_copy_source_session.cc:205] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:18.793720 11070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f44b2b53de84a87a8757c0d00bfed37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.810346 11061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f44b2b53de84a87a8757c0d00bfed37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.810761 11083 tablet_copy_client.cc:783] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:18.811164 11083 tablet_copy_client.cc:647] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:18.814399 11083 tablet_copy_client.cc:523] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:18.819718 11083 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap starting.
I20240503 02:13:18.835004 11083 log.cc:824] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:18.841291  8101 consensus_queue.cc:237] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:18.869886  8362 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.869998 11070 tablet_copy_client.cc:783] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:18.870467 11070 tablet_copy_client.cc:647] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
W20240503 02:13:18.870862  8038 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72e2bb75da654e8ab45bba8ac077f28c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:18.871137 10942 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.874783 11070 tablet_copy_client.cc:523] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:18.879747  8102 consensus_queue.cc:237] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:18.880061  7969 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.880995 10942 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.905884 11119 ts_tablet_manager.cc:902] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:18.917102 10319 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:18.919203 11070 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
W20240503 02:13:18.919528  8038 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeaee03abc644807b8278cfb78e4f02b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:18.920305  8361 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.921000 10989 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.926556  7823 catalog_manager.cc:5484] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:18.929705  8232 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.936760 10942 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:18.956027  7971 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:18.957082 11061 tablet_copy_client.cc:783] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:18.957584 11061 tablet_copy_client.cc:647] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:18.970682 11119 tablet_copy_client.cc:308] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:37681
I20240503 02:13:18.971418  8360 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:18.972755 11061 tablet_copy_client.cc:523] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:18.983907  8361 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:18.991873  7829 catalog_manager.cc:5484] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:18.996397 11136 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.997784 11127 ts_tablet_manager.cc:902] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:19.023723 11147 ts_tablet_manager.cc:902] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:19.058059 11061 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap starting.
I20240503 02:13:19.095743 11070 log.cc:824] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:19.100235 11147 tablet_copy_client.cc:308] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:19.099800  8232 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:19.101172  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet b64f66ace86e42f3ac213ba85bcab584 from peer ca05c9a8bee148dab8a2a982b8fa2ea4 ({username='slave'} at 127.0.145.8:52015)
I20240503 02:13:19.101423  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet b64f66ace86e42f3ac213ba85bcab584 from peer ca05c9a8bee148dab8a2a982b8fa2ea4 at {username='slave'} at 127.0.145.8:52015: session id = ca05c9a8bee148dab8a2a982b8fa2ea4-b64f66ace86e42f3ac213ba85bcab584
I20240503 02:13:19.104275 11163 ts_tablet_manager.cc:902] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512: Initiating tablet copy from peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:19.104849  8122 tablet_copy_source_session.cc:205] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:19.105674  7991 tablet_copy_service.cc:140] P d986fcfb926e47baa7d2d5b99f697334: Received BeginTabletCopySession request for tablet 4f2cedba9f9a4e709623fceeffcfd095 from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:34517)
I20240503 02:13:19.105957  7991 tablet_copy_service.cc:161] P d986fcfb926e47baa7d2d5b99f697334: Beginning new tablet copy session on tablet 4f2cedba9f9a4e709623fceeffcfd095 from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:34517: session id = d826d4dae2aa4e05b2f64c9b17a1126f-4f2cedba9f9a4e709623fceeffcfd095
I20240503 02:13:19.106535 11147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64f66ace86e42f3ac213ba85bcab584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.109876  7991 tablet_copy_source_session.cc:205] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:19.115677 11147 tablet_copy_client.cc:783] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:19.116056 11147 tablet_copy_client.cc:647] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:19.120117 11147 tablet_copy_client.cc:523] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:19.123962 11147 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap starting.
I20240503 02:13:19.131747 11119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2cedba9f9a4e709623fceeffcfd095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.150961  8362 consensus_queue.cc:237] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:19.165957 11127 tablet_copy_client.cc:308] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:19.170035  8361 consensus_queue.cc:237] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:19.179435  8102 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:19.180347 10592 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:19.189296 11147 log.cc:824] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:19.195551  7969 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:19.199174 10592 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:19.214089  8231 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:19.214778 10592 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:19.217615  8295 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e4acc8b5ceb4cd5b0f3708ff4f19298. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:19.227077 11070 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:19.228251 11070 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:19.228828 11163 tablet_copy_client.cc:308] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:37681
I20240503 02:13:19.229557 11070 ts_tablet_manager.cc:1367] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.311s	user 0.068s	sys 0.009s
I20240503 02:13:19.242095 10592 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:19.244709  7991 tablet_copy_service.cc:140] P d986fcfb926e47baa7d2d5b99f697334: Received BeginTabletCopySession request for tablet 5ea56405515a40f785726bd58ec3d26a from peer b85e85e631ab4e04a86bfe5a9f0aa512 ({username='slave'} at 127.0.145.6:46235)
I20240503 02:13:19.244923  7991 tablet_copy_service.cc:161] P d986fcfb926e47baa7d2d5b99f697334: Beginning new tablet copy session on tablet 5ea56405515a40f785726bd58ec3d26a from peer b85e85e631ab4e04a86bfe5a9f0aa512 at {username='slave'} at 127.0.145.6:46235: session id = b85e85e631ab4e04a86bfe5a9f0aa512-5ea56405515a40f785726bd58ec3d26a
I20240503 02:13:19.247962  7991 tablet_copy_source_session.cc:205] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:19.251591 11163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea56405515a40f785726bd58ec3d26a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.257086  7825 catalog_manager.cc:5484] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:19.258852 11070 raft_consensus.cc:348] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:19.259460 11070 raft_consensus.cc:729] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:19.260108 11070 consensus_queue.cc:260] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:19.262980  8101 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:19.263399  8297 consensus_peers.cc:485] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de9c2a3396864696af88be1ca12ee0f1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:19.267297  8231 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:19.273985 10939 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:19.274447 10995 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:19.282657  7829 catalog_manager.cc:5484] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:19.300371 11119 tablet_copy_client.cc:783] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:19.300870 11119 tablet_copy_client.cc:647] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:19.304476  7971 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:19.307781 11119 tablet_copy_client.cc:523] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:19.314682 11163 tablet_copy_client.cc:783] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:19.314736 11119 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
I20240503 02:13:19.316684 11163 tablet_copy_client.cc:647] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:19.319505  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet f14607425da84bb4b162a5789daa844c from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:54377)
I20240503 02:13:19.319818  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet f14607425da84bb4b162a5789daa844c from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:54377: session id = d826d4dae2aa4e05b2f64c9b17a1126f-f14607425da84bb4b162a5789daa844c
I20240503 02:13:19.321465 11163 tablet_copy_client.cc:523] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:19.324035  8252 tablet_copy_source_session.cc:205] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:19.330696 11127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14607425da84bb4b162a5789daa844c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.340260 11127 tablet_copy_client.cc:783] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:19.340651 11127 tablet_copy_client.cc:647] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:19.347337 11127 tablet_copy_client.cc:523] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:19.348310  8100 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:19.354259 11163 tablet_bootstrap.cc:492] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap starting.
I20240503 02:13:19.354919 11196 ts_tablet_manager.cc:902] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:19.368768 11127 tablet_bootstrap.cc:492] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
I20240503 02:13:19.376658 11119 log.cc:824] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:19.391544  8361 consensus_queue.cc:237] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:19.393899 11196 tablet_copy_client.cc:308] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:19.393554  8101 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:19.394981  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet aeaee03abc644807b8278cfb78e4f02b from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:48749)
I20240503 02:13:19.395171  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet aeaee03abc644807b8278cfb78e4f02b from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:48749: session id = 03f99420df704238ab300ad6e58d878c-aeaee03abc644807b8278cfb78e4f02b
I20240503 02:13:19.398823  8122 tablet_copy_source_session.cc:205] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:19.400559 11196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeaee03abc644807b8278cfb78e4f02b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:13:19.411939  8295 consensus_peers.cc:485] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3d4b912095244de816d3089a31b8649. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:19.413218  7969 raft_consensus.cc:1261] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:19.413926 10592 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:19.416121 11070 ts_tablet_manager.cc:1398] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.186s	user 0.019s	sys 0.033s
I20240503 02:13:19.417964 11196 tablet_copy_client.cc:783] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:19.418359 11196 tablet_copy_client.cc:647] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:19.420711  8231 raft_consensus.cc:1261] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:19.421515 10592 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:19.426381 10995 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:19.429502  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-0f44b2b53de84a87a8757c0d00bfed37 received from {username='slave'} at 127.0.145.5:48749
I20240503 02:13:19.429674  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session 03f99420df704238ab300ad6e58d878c-0f44b2b53de84a87a8757c0d00bfed37 on tablet 0f44b2b53de84a87a8757c0d00bfed37 with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:19.430017 11196 tablet_copy_client.cc:523] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:19.437705  8231 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:19.439795 11223 ts_tablet_manager.cc:902] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:19.452504  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:19.475030 11196 tablet_bootstrap.cc:492] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
I20240503 02:13:19.478303  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:19.480345  7829 catalog_manager.cc:5484] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:19.466245  7969 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:19.501794 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:19.507678 11223 tablet_copy_client.cc:308] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:19.508836  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 72e2bb75da654e8ab45bba8ac077f28c from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:48749)
I20240503 02:13:19.509030  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 72e2bb75da654e8ab45bba8ac077f28c from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:48749: session id = 03f99420df704238ab300ad6e58d878c-72e2bb75da654e8ab45bba8ac077f28c
I20240503 02:13:19.512343  8122 tablet_copy_source_session.cc:205] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:19.513880 11223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e2bb75da654e8ab45bba8ac077f28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.524605  8231 consensus_queue.cc:237] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:19.530725 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:19.557006  7970 consensus_queue.cc:237] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:19.562124  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:19.593214  8101 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:19.593937 11198 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:19.603921  8361 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:19.604984 11198 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:19.609859 11061 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:19.615783  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:13:19.626906  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32e527fa9b2f441cb5668435681b4ff6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:19.629855  8362 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:19.630709  7907 consensus_peers.cc:485] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cab70b62bdc747a9b653bffde680f37e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:19.631062 10659 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:19.632401  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:19.632653 11198 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:19.637501  8102 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:19.639010  8100 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:19.639816 10659 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:19.643326  7822 catalog_manager.cc:5484] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:19.648605 11223 tablet_copy_client.cc:783] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:19.648912 11223 tablet_copy_client.cc:647] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:19.651489  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:19.653368  8360 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:19.655233 11223 tablet_copy_client.cc:523] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:19.697533 11257 ts_tablet_manager.cc:902] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:19.700448 11223 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
I20240503 02:13:19.703992  8101 consensus_queue.cc:237] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
W20240503 02:13:19.708848 11253 log.cc:924] Time spent T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20240503 02:13:19.710253 10659 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:19.717799  8102 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:19.725719  7822 catalog_manager.cc:5484] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:19.727659 11257 tablet_copy_client.cc:308] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:19.610721 11061 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap complete.
I20240503 02:13:19.744823  8362 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:19.724179 11147 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:19.747959 11147 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap complete.
I20240503 02:13:19.748257 11147 ts_tablet_manager.cc:1367] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent bootstrapping tablet: real 0.624s	user 0.071s	sys 0.000s
I20240503 02:13:19.763769  8231 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:19.764353 11136 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:19.766911  7969 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:19.767516 11141 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:19.768142 10319 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:19.772524  8231 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:19.779400  7823 catalog_manager.cc:5484] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:19.745262 11061 ts_tablet_manager.cc:1367] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent bootstrapping tablet: real 0.688s	user 0.055s	sys 0.009s
I20240503 02:13:19.814363  8101 consensus_queue.cc:237] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:19.824249  8362 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:19.829254 10319 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:13:19.831310  8038 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:19.836155 11136 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:19.852499 11163 tablet_bootstrap.cc:492] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:19.852844  8362 consensus_queue.cc:237] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:19.855361 11163 tablet_bootstrap.cc:492] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap complete.
I20240503 02:13:19.855938 11163 ts_tablet_manager.cc:1367] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent bootstrapping tablet: real 0.502s	user 0.045s	sys 0.018s
I20240503 02:13:19.863968  8101 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:19.864099  8361 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:19.870122 11163 raft_consensus.cc:348] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:19.870752 11163 raft_consensus.cc:729] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Initialized, Role: LEARNER
I20240503 02:13:19.871948  7825 catalog_manager.cc:5484] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 4 to 5, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:19.876048 11061 raft_consensus.cc:348] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:19.876821 11061 raft_consensus.cc:729] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Initialized, Role: LEARNER
I20240503 02:13:19.877447 11061 consensus_queue.cc:260] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:19.881867 11147 raft_consensus.cc:348] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:19.883478 11147 raft_consensus.cc:729] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Initialized, Role: LEARNER
I20240503 02:13:19.884994 11147 consensus_queue.cc:260] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:19.892416 11061 ts_tablet_manager.cc:1398] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent starting tablet: real 0.099s	user 0.011s	sys 0.004s
I20240503 02:13:19.871389 11163 consensus_queue.cc:260] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:19.895704 11147 ts_tablet_manager.cc:1398] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent starting tablet: real 0.147s	user 0.005s	sys 0.000s
I20240503 02:13:19.896456  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-0f44b2b53de84a87a8757c0d00bfed37 received from {username='slave'} at 127.0.145.8:52015
I20240503 02:13:19.896626  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-0f44b2b53de84a87a8757c0d00bfed37 on tablet 0f44b2b53de84a87a8757c0d00bfed37 with peer ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:19.899881  8231 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:19.904809  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-b64f66ace86e42f3ac213ba85bcab584 received from {username='slave'} at 127.0.145.8:52015
I20240503 02:13:19.904976  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-b64f66ace86e42f3ac213ba85bcab584 on tablet b64f66ace86e42f3ac213ba85bcab584 with peer ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:19.905711 11289 ts_tablet_manager.cc:902] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:19.906674 10261 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:19.907533 11136 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:19.861632  7969 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:19.913444 11163 ts_tablet_manager.cc:1398] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent starting tablet: real 0.057s	user 0.014s	sys 0.002s
I20240503 02:13:19.914549  7991 tablet_copy_service.cc:342] P d986fcfb926e47baa7d2d5b99f697334: Request end of tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-5ea56405515a40f785726bd58ec3d26a received from {username='slave'} at 127.0.145.6:46235
I20240503 02:13:19.914750  7991 tablet_copy_service.cc:433] P d986fcfb926e47baa7d2d5b99f697334: ending tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-5ea56405515a40f785726bd58ec3d26a on tablet 5ea56405515a40f785726bd58ec3d26a with peer b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:19.917817 11288 ts_tablet_manager.cc:902] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:19.920222  7970 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:19.920608 10319 raft_consensus.cc:1013] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: attempt to promote peer ca05c9a8bee148dab8a2a982b8fa2ea4: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 02:13:19.926375  9863 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:13:19.927990  8038 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f44b2b53de84a87a8757c0d00bfed37. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:19.928850  8038 consensus_peers.cc:485] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4af5884849ec4ac3891ebf0f08ee2cff. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:19.931574  9996 raft_consensus.cc:1203] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:19.934172 11136 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:19.939955  8360 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:19.944590  8231 raft_consensus.cc:1261] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:19.945833 11127 tablet_bootstrap.cc:492] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:19.946592 11127 tablet_bootstrap.cc:492] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:13:19.947187 11127 ts_tablet_manager.cc:1367] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 0.579s	user 0.067s	sys 0.000s
I20240503 02:13:19.947180 11227 consensus_queue.cc:1035] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
W20240503 02:13:19.948426  8297 consensus_peers.cc:485] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3dbfc7f12fc142f98b1d1c026fd49f5a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:19.952773  8232 consensus_queue.cc:237] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:19.960332 11136 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:19.960897 11127 raft_consensus.cc:348] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:19.962196  8100 raft_consensus.cc:1261] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:19.962235 11127 raft_consensus.cc:729] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:13:19.963466 11227 consensus_queue.cc:1035] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:19.965881 10260 raft_consensus.cc:1203] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:19.978693 11288 tablet_copy_client.cc:308] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:19.979593 11227 raft_consensus.cc:2932] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:19.981791  7970 raft_consensus.cc:1261] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:19.982650 11198 consensus_queue.cc:1035] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:19.982025  8231 raft_consensus.cc:2932] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:20.000912 11127 consensus_queue.cc:260] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:20.001250 11163 ts_tablet_manager.cc:902] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:20.000967  7823 catalog_manager.cc:5484] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:20.028568  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:43609)
I20240503 02:13:20.028862  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:43609: session id = d826d4dae2aa4e05b2f64c9b17a1126f-7e4acc8b5ceb4cd5b0f3708ff4f19298
I20240503 02:13:20.032606  8382 tablet_copy_source_session.cc:205] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:20.036861 11257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:20.042948  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet d3d4b912095244de816d3089a31b8649 from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:43609)
I20240503 02:13:20.043308  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet d3d4b912095244de816d3089a31b8649 from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:43609: session id = d826d4dae2aa4e05b2f64c9b17a1126f-d3d4b912095244de816d3089a31b8649
I20240503 02:13:20.045460 11127 ts_tablet_manager.cc:1398] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.098s	user 0.020s	sys 0.005s
I20240503 02:13:20.046763  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-f14607425da84bb4b162a5789daa844c received from {username='slave'} at 127.0.145.7:54377
I20240503 02:13:20.047021  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-f14607425da84bb4b162a5789daa844c on tablet f14607425da84bb4b162a5789daa844c with peer d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:20.059307 11257 tablet_copy_client.cc:783] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:20.059701 11257 tablet_copy_client.cc:647] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:20.067951 11257 tablet_copy_client.cc:523] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:20.068176  8382 tablet_copy_source_session.cc:205] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:20.069797 11288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d4b912095244de816d3089a31b8649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:20.082098 11288 tablet_copy_client.cc:783] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:20.082715 11288 tablet_copy_client.cc:647] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:20.091233 10319 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.094120 11196 tablet_bootstrap.cc:492] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:20.095588  8231 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:20.097555 11223 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:20.098026 11223 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:20.098315 11223 ts_tablet_manager.cc:1367] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.398s	user 0.049s	sys 0.005s
I20240503 02:13:20.099664 11223 raft_consensus.cc:348] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:20.100011 11223 raft_consensus.cc:729] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:20.100236 11223 consensus_queue.cc:260] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:20.101487 11223 ts_tablet_manager.cc:1398] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:13:20.101552 11119 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:20.102022 11119 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:13:20.102303 11119 ts_tablet_manager.cc:1367] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 0.788s	user 0.076s	sys 0.008s
I20240503 02:13:20.102342  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-72e2bb75da654e8ab45bba8ac077f28c received from {username='slave'} at 127.0.145.5:48749
I20240503 02:13:20.102475  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session 03f99420df704238ab300ad6e58d878c-72e2bb75da654e8ab45bba8ac077f28c on tablet 72e2bb75da654e8ab45bba8ac077f28c with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:20.104789 11196 tablet_bootstrap.cc:492] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:20.105091 11196 ts_tablet_manager.cc:1367] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.630s	user 0.042s	sys 0.012s
I20240503 02:13:20.105855 11196 raft_consensus.cc:348] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:20.106129 11196 raft_consensus.cc:729] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:20.106318 11196 consensus_queue.cc:260] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:20.105841 11134 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.106882 11196 ts_tablet_manager.cc:1398] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:13:20.107795  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-aeaee03abc644807b8278cfb78e4f02b received from {username='slave'} at 127.0.145.5:48749
I20240503 02:13:20.107959  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session 03f99420df704238ab300ad6e58d878c-aeaee03abc644807b8278cfb78e4f02b on tablet aeaee03abc644807b8278cfb78e4f02b with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:20.106382 11119 raft_consensus.cc:348] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:20.109627 11119 raft_consensus.cc:729] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:13:20.109932 11119 consensus_queue.cc:260] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:20.110599 11119 ts_tablet_manager.cc:1398] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.000s
I20240503 02:13:20.112524 10260 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.114694  7991 tablet_copy_service.cc:342] P d986fcfb926e47baa7d2d5b99f697334: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-4f2cedba9f9a4e709623fceeffcfd095 received from {username='slave'} at 127.0.145.7:34517
I20240503 02:13:20.114871  7991 tablet_copy_service.cc:433] P d986fcfb926e47baa7d2d5b99f697334: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-4f2cedba9f9a4e709623fceeffcfd095 on tablet 4f2cedba9f9a4e709623fceeffcfd095 with peer d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:20.122084 11136 raft_consensus.cc:1050] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 to VOTER
I20240503 02:13:20.122869 11288 tablet_copy_client.cc:523] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:20.126730  7823 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 5 to 6, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.139683 11136 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:20.140321  8102 raft_consensus.cc:1261] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:20.149365 11254 consensus_queue.cc:1035] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:20.168294 11198 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:20.170305 11317 ts_tablet_manager.cc:902] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512: Initiating tablet copy from peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:20.184971  7970 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:20.191154  7825 catalog_manager.cc:5484] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:20.193945  8038 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f44b2b53de84a87a8757c0d00bfed37. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.194380  8101 raft_consensus.cc:2932] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:20.205762 10260 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:20.206418  8362 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:20.206590  9864 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:20.206696 11136 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:20.207785 10319 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:20.208276 10319 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:20.209873  8362 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.209798  9864 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.211475 11257 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
I20240503 02:13:20.215906  7969 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:20.221163 11288 tablet_bootstrap.cc:492] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
I20240503 02:13:20.238052 10127 raft_consensus.cc:1203] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:20.247027 10127 raft_consensus.cc:1203] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:20.255097 11136 raft_consensus.cc:1013] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: attempt to promote peer 03f99420df704238ab300ad6e58d878c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 02:13:20.255227  8231 consensus_queue.cc:237] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:20.255292 11136 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.259308  8361 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.260296 10127 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.261744  7970 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.259348  8362 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.263358 11339 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.263774 11254 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.264436 11045 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.253522 11317 tablet_copy_client.cc:308] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:37681
I20240503 02:13:20.277041  7991 tablet_copy_service.cc:140] P d986fcfb926e47baa7d2d5b99f697334: Received BeginTabletCopySession request for tablet cab70b62bdc747a9b653bffde680f37e from peer b85e85e631ab4e04a86bfe5a9f0aa512 ({username='slave'} at 127.0.145.6:46235)
I20240503 02:13:20.277287  7991 tablet_copy_service.cc:161] P d986fcfb926e47baa7d2d5b99f697334: Beginning new tablet copy session on tablet cab70b62bdc747a9b653bffde680f37e from peer b85e85e631ab4e04a86bfe5a9f0aa512 at {username='slave'} at 127.0.145.6:46235: session id = b85e85e631ab4e04a86bfe5a9f0aa512-cab70b62bdc747a9b653bffde680f37e
I20240503 02:13:20.276984  7822 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 6 to 7, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.281785  7991 tablet_copy_source_session.cc:205] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 02:13:20.289852  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ceae1c7c45a34b24badbbe23cb9e0176. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:20.290208  8168 consensus_peers.cc:485] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f14607425da84bb4b162a5789daa844c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.295033 11289 tablet_copy_client.cc:308] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:20.300391 11317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cab70b62bdc747a9b653bffde680f37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:20.309665 10260 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.313834  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet de9c2a3396864696af88be1ca12ee0f1 from peer ca05c9a8bee148dab8a2a982b8fa2ea4 ({username='slave'} at 127.0.145.8:38819)
I20240503 02:13:20.314182  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet de9c2a3396864696af88be1ca12ee0f1 from peer ca05c9a8bee148dab8a2a982b8fa2ea4 at {username='slave'} at 127.0.145.8:38819: session id = ca05c9a8bee148dab8a2a982b8fa2ea4-de9c2a3396864696af88be1ca12ee0f1
I20240503 02:13:20.317641  8382 tablet_copy_source_session.cc:205] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:20.317534 11045 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:20.319617 11289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de9c2a3396864696af88be1ca12ee0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:20.326432  9863 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:20.328640  7823 catalog_manager.cc:5484] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 4 to 5, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:20.329041  7969 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:20.334096 11289 tablet_copy_client.cc:783] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:20.334911 11289 tablet_copy_client.cc:647] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:20.337467 11289 tablet_copy_client.cc:523] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:20.341228 11289 tablet_bootstrap.cc:492] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap starting.
W20240503 02:13:20.342001  8035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.1:37681, user_credentials={real_user=slave}} blocked reactor thread for 48818.1us
I20240503 02:13:20.341670  9864 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.342344 11136 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:13:20.343470  8035 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeaee03abc644807b8278cfb78e4f02b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.346830  9862 raft_consensus.cc:1203] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:20.348379  7969 raft_consensus.cc:858] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
W20240503 02:13:20.348759  7969 raft_consensus.cc:1565] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:20.350783  8101 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:20.356739 11136 raft_consensus.cc:1050] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER 03f99420df704238ab300ad6e58d878c to VOTER
I20240503 02:13:20.357839 11136 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
W20240503 02:13:20.362190  8038 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f44b2b53de84a87a8757c0d00bfed37. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.362700  9864 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:20.363322  8361 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:20.363437 11136 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:20.363958 10319 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:20.367595 10319 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.372593  9864 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.374177  8361 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.375873  7971 consensus_queue.cc:237] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:20.377490 11349 ts_tablet_manager.cc:902] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:20.382855 11163 tablet_copy_client.cc:308] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:20.388716  7822 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c reported cstate change: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.393965 11196 ts_tablet_manager.cc:902] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:20.398376 10260 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:20.411634 11136 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:20.413631 10261 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.416873  8231 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:20.420931 11196 tablet_copy_client.cc:308] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:20.426124 11198 raft_consensus.cc:1050] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f to VOTER
I20240503 02:13:20.427309 11198 consensus_queue.cc:237] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:20.431558  7968 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:20.432396 11254 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:13:20.438022  8168 consensus_peers.cc:485] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f14607425da84bb4b162a5789daa844c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.441967  8361 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:20.442838 11198 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.445281 11119 ts_tablet_manager.cc:902] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:20.449429  8362 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:20.455471  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet ceae1c7c45a34b24badbbe23cb9e0176 from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:37535)
I20240503 02:13:20.455751  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet ceae1c7c45a34b24badbbe23cb9e0176 from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:37535: session id = 03f99420df704238ab300ad6e58d878c-ceae1c7c45a34b24badbbe23cb9e0176
I20240503 02:13:20.457587 10128 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:20.458870 10128 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.459549 10659 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.459712  8361 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:20.460428 11045 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.461037 11045 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:20.460561  8252 tablet_copy_source_session.cc:205] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:20.467525 11196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceae1c7c45a34b24badbbe23cb9e0176. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:20.469238 11200 raft_consensus.cc:1013] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: attempt to promote peer d826d4dae2aa4e05b2f64c9b17a1126f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 02:13:20.473686 11200 raft_consensus.cc:1050] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334: attempting to promote NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 to VOTER
I20240503 02:13:20.474730 11200 consensus_queue.cc:237] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:20.466776 10127 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:20.476354  8231 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.475946  7969 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.477012 10659 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.477358 11196 tablet_copy_client.cc:783] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:20.477608 11196 tablet_copy_client.cc:647] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:20.480891 11196 tablet_copy_client.cc:523] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:20.483981  8360 raft_consensus.cc:1261] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.484850 11200 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:13:20.498854 11350 log.cc:924] Time spent T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20240503 02:13:20.500015  7971 consensus_queue.cc:237] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:20.502565  8101 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:20.505033 11198 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:20.437816 11083 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:20.510737 11083 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap complete.
I20240503 02:13:20.510998 11289 tablet_bootstrap.cc:492] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:20.511132 11083 ts_tablet_manager.cc:1367] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent bootstrapping tablet: real 1.692s	user 0.120s	sys 0.095s
I20240503 02:13:20.511499 11289 tablet_bootstrap.cc:492] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap complete.
I20240503 02:13:20.510735  8362 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:20.511870 11289 ts_tablet_manager.cc:1367] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent bootstrapping tablet: real 0.171s	user 0.057s	sys 0.013s
I20240503 02:13:20.484766  9996 raft_consensus.cc:1261] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.469574 11349 tablet_copy_client.cc:308] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:20.512720 11289 raft_consensus.cc:348] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:20.513031 11289 raft_consensus.cc:729] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Initialized, Role: LEARNER
I20240503 02:13:20.513491 11289 consensus_queue.cc:260] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:20.514572 11289 ts_tablet_manager.cc:1398] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:13:20.513692 11200 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 02:13:20.515625  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-de9c2a3396864696af88be1ca12ee0f1 received from {username='slave'} at 127.0.145.8:38819
I20240503 02:13:20.515789  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-de9c2a3396864696af88be1ca12ee0f1 on tablet de9c2a3396864696af88be1ca12ee0f1 with peer ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:20.518003  7822 catalog_manager.cc:5484] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:20.522807 11045 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:20.528311 10659 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:20.529188 11196 tablet_bootstrap.cc:492] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
I20240503 02:13:20.529982  8361 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:20.532696 11339 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:20.533578  8101 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.538084 11289 ts_tablet_manager.cc:902] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:20.540445  7823 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:20.547050  8231 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:20.555692  8102 raft_consensus.cc:1261] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.559633  7823 catalog_manager.cc:5484] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 4 to 5, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:20.575318  7968 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:20.509865 11317 tablet_copy_client.cc:783] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:20.577519 11317 tablet_copy_client.cc:647] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:20.583631 11317 tablet_copy_client.cc:523] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:20.587842 11317 tablet_bootstrap.cc:492] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap starting.
W20240503 02:13:20.588320  8167 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e966caa651e4e42890be5d79f851a1f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.592311  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet 32e527fa9b2f441cb5668435681b4ff6 from peer b85e85e631ab4e04a86bfe5a9f0aa512 ({username='slave'} at 127.0.145.6:56443)
I20240503 02:13:20.592494  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet 32e527fa9b2f441cb5668435681b4ff6 from peer b85e85e631ab4e04a86bfe5a9f0aa512 at {username='slave'} at 127.0.145.6:56443: session id = b85e85e631ab4e04a86bfe5a9f0aa512-32e527fa9b2f441cb5668435681b4ff6
I20240503 02:13:20.595661  8252 tablet_copy_source_session.cc:205] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:20.599292 11200 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.599998 11200 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.610927  8232 raft_consensus.cc:1261] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:20.613113 11119 tablet_copy_client.cc:308] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:20.619959  8101 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:20.632166 11324 raft_consensus.cc:1050] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 to VOTER
I20240503 02:13:20.633311 11392 consensus_queue.cc:1035] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:20.633237 11324 consensus_queue.cc:237] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:20.638104  7970 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:20.640043 11196 tablet_bootstrap.cc:700] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c: Time spent opening tablet: real 0.110s	user 0.003s	sys 0.000s
I20240503 02:13:20.657542  8232 consensus_queue.cc:237] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:20.657914 10127 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:20.663398  8361 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:20.664260 11254 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:20.597131 11163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e527fa9b2f441cb5668435681b4ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:20.684300 11163 tablet_copy_client.cc:783] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:20.684741 11163 tablet_copy_client.cc:647] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:20.597692 11083 raft_consensus.cc:348] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:20.685098 11083 raft_consensus.cc:729] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Initialized, Role: LEARNER
I20240503 02:13:20.685494 11083 consensus_queue.cc:260] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:20.692835 11163 tablet_copy_client.cc:523] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:20.696648 11163 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap starting.
I20240503 02:13:20.699293 10126 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:20.711901 11397 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:20.725149  8102 consensus_queue.cc:237] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:20.708920 10319 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.055s
I20240503 02:13:20.749666  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet aeaee03abc644807b8278cfb78e4f02b from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:59845)
I20240503 02:13:20.749912  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet aeaee03abc644807b8278cfb78e4f02b from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:59845: session id = d826d4dae2aa4e05b2f64c9b17a1126f-aeaee03abc644807b8278cfb78e4f02b
I20240503 02:13:20.753118  8122 tablet_copy_source_session.cc:205] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:20.755307 11119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeaee03abc644807b8278cfb78e4f02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:20.761365 11083 ts_tablet_manager.cc:1398] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent starting tablet: real 0.250s	user 0.001s	sys 0.003s
I20240503 02:13:20.765033 10260 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.765895 11141 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.781143 10128 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:20.782325 11254 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:20.787081 10261 raft_consensus.cc:1203] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:20.788725 11254 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:20.792001 10128 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:20.797524  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 4af5884849ec4ac3891ebf0f08ee2cff from peer b85e85e631ab4e04a86bfe5a9f0aa512 ({username='slave'} at 127.0.145.6:56611)
I20240503 02:13:20.797725  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 4af5884849ec4ac3891ebf0f08ee2cff from peer b85e85e631ab4e04a86bfe5a9f0aa512 at {username='slave'} at 127.0.145.6:56611: session id = b85e85e631ab4e04a86bfe5a9f0aa512-4af5884849ec4ac3891ebf0f08ee2cff
I20240503 02:13:20.800774  8122 tablet_copy_source_session.cc:205] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:20.802229 11349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af5884849ec4ac3891ebf0f08ee2cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:20.802542  8361 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.802750  8231 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.803498 10319 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.801570  7825 catalog_manager.cc:5484] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:20.806329 11136 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.810181  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f14607425da84bb4b162a5789daa844c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:20.810986  8101 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:20.812301  8362 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:20.814438 10261 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:20.815177 10319 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:20.815348  9864 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:20.815394  8360 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:20.817548 11349 tablet_copy_client.cc:783] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:20.817871 11349 tablet_copy_client.cc:647] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:20.817891  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-80b4e6aa20f64473be8bcbf47baf66e4 received from {username='slave'} at 127.0.145.6:49885
I20240503 02:13:20.818012  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-80b4e6aa20f64473be8bcbf47baf66e4 on tablet 80b4e6aa20f64473be8bcbf47baf66e4 with peer b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:20.819660 11136 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:20.820185 11136 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:20.823441 11136 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:20.827454  9864 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
W20240503 02:13:20.830396  7907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.6:40397, user_credentials={real_user=slave}} blocked reactor thread for 116138us
I20240503 02:13:20.831878  8360 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:20.832033  9995 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:20.834182  8359 consensus_queue.cc:237] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:20.836324 11119 tablet_copy_client.cc:783] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:20.837435 11119 tablet_copy_client.cc:647] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:20.839399  7823 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c reported cstate change: config changed from index 8 to 9, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) evicted. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:20.840994  7970 raft_consensus.cc:858] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
W20240503 02:13:20.841163  7970 raft_consensus.cc:1565] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 8' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:20.841149  7827 catalog_manager.cc:5484] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:20.841502  8362 raft_consensus.cc:1261] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:20.842164  7971 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:20.841697  7970 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.842640  8230 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:20.842711 11227 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:20.843128  7906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.2:35309, user_credentials={real_user=slave}} blocked reactor thread for 89178.1us
I20240503 02:13:20.843997 11349 tablet_copy_client.cc:523] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:20.844586 11227 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:20.846711  7907 consensus_peers.cc:485] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f231fe5dfd9e4c87b90bad75818a7060. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.847426 11401 consensus_queue.cc:1035] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:20.850889  7969 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:20.852500 11386 ts_tablet_manager.cc:902] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:20.860402 10995 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.861368  8230 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.861665 11198 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:20.866106  7967 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
W20240503 02:13:20.875984  7906 consensus_peers.cc:485] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f2cedba9f9a4e709623fceeffcfd095. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.879033  7967 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 02:13:20.880004  8038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.5:44891, user_credentials={real_user=slave}} blocked reactor thread for 44469.2us
I20240503 02:13:20.882815 11134 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:20.883162 11401 raft_consensus.cc:1050] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334: attempting to promote NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f to VOTER
I20240503 02:13:20.883863 10319 raft_consensus.cc:1050] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER 03f99420df704238ab300ad6e58d878c to VOTER
I20240503 02:13:20.884192 11401 consensus_queue.cc:237] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:20.884774 10319 consensus_queue.cc:237] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:20.887645  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f44b2b53de84a87a8757c0d00bfed37 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:20.888283  8229 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:20.889421 10261 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:20.891316 11397 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:20.892561  7968 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
W20240503 02:13:20.899827  7906 consensus_peers.cc:485] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f2cedba9f9a4e709623fceeffcfd095. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.899909  7822 catalog_manager.cc:5484] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.902276 10127 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:20.904115 11400 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:20.905117 11400 raft_consensus.cc:2932] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.914660  8362 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:20.918268 10128 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
W20240503 02:13:20.923131  8298 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c23b06764ce54acb8388de94aea4cd4b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.926505 11119 tablet_copy_client.cc:523] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:20.928179 10659 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:20.924252  8359 raft_consensus.cc:2932] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.930284 11119 tablet_bootstrap.cc:492] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
I20240503 02:13:20.932024  9864 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:20.933046 11134 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:20.934175 11349 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap starting.
I20240503 02:13:20.934167  7971 raft_consensus.cc:858] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }.
  Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }: 
  New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:20.934386  8360 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:13:20.934418  7971 raft_consensus.cc:1565] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }.
  Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }: 
  New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:20.935221 11134 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:20.935518  7971 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.930366  8232 raft_consensus.cc:2932] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:20.941870  7967 consensus_queue.cc:237] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:20.944088  7825 catalog_manager.cc:5484] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:20.945091  8098 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:20.946974  8358 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.947677  9864 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.948647 11134 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.949168 11136 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.952342  8229 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:20.955550 11289 tablet_copy_client.cc:308] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:20.956400  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet 3dbfc7f12fc142f98b1d1c026fd49f5a from peer ca05c9a8bee148dab8a2a982b8fa2ea4 ({username='slave'} at 127.0.145.8:38819)
I20240503 02:13:20.956588  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet 3dbfc7f12fc142f98b1d1c026fd49f5a from peer ca05c9a8bee148dab8a2a982b8fa2ea4 at {username='slave'} at 127.0.145.8:38819: session id = ca05c9a8bee148dab8a2a982b8fa2ea4-3dbfc7f12fc142f98b1d1c026fd49f5a
I20240503 02:13:20.960489  8382 tablet_copy_source_session.cc:205] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:20.960614  8358 raft_consensus.cc:1261] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:20.961241 11401 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:20.962101 11289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dbfc7f12fc142f98b1d1c026fd49f5a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:13:20.963776  8035 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:20.964049  8035 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72e2bb75da654e8ab45bba8ac077f28c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:20.963749 11392 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:20.964439 11134 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:20.964956 10260 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:20.965721  7968 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:20.969667  8230 consensus_queue.cc:237] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:20.970463  8358 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:20.971256 10319 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:20.975339  9995 raft_consensus.cc:1261] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:20.952862  7829 catalog_manager.cc:5484] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:20.976907  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5ea56405515a40f785726bd58ec3d26a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:20.977414 11401 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:20.981251  9995 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:20.983759  9864 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:20.985334 11134 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:20.985994 11141 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:20.995082  7829 catalog_manager.cc:5484] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:20.996755  7823 catalog_manager.cc:5484] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 5 to 6, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:20.997113 11134 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:20.999565 11324 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:21.000761  8101 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:21.003262 10261 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.006047  7971 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:21.011499  7827 catalog_manager.cc:5484] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.012795  8361 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.014133  9862 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.019516  8362 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.028476 10319 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.033445 10128 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:21.034193 11198 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:21.037245 11198 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:21.039517  9844 tablet_service.cc:1514] Processing DeleteTablet for tablet f14607425da84bb4b162a5789daa844c with delete_type TABLET_DATA_TOMBSTONED (TS 03f99420df704238ab300ad6e58d878c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35050
I20240503 02:13:21.042701  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f14607425da84bb4b162a5789daa844c with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:21.043592 10128 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:21.052286 11289 tablet_copy_client.cc:783] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:21.052662 11289 tablet_copy_client.cc:647] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:21.058089  7828 catalog_manager.cc:5484] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: config changed from index 7 to 8, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:21.058483  8099 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:21.059715 10659 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:21.061496  8099 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
W20240503 02:13:21.066793  7817 catalog_manager.cc:4797] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): delete failed for tablet f14607425da84bb4b162a5789daa844c because the tablet was not found. No further retry: Not found: Tablet not found: f14607425da84bb4b162a5789daa844c
I20240503 02:13:21.067201 11134 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.072145  8360 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.018308  8231 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:21.082578  9864 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.089313 11289 tablet_copy_client.cc:523] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:21.090698  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f44b2b53de84a87a8757c0d00bfed37 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:21.093202 11289 tablet_bootstrap.cc:492] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap starting.
I20240503 02:13:21.096103  9862 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.102041  8081 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea56405515a40f785726bd58ec3d26a with delete_type TABLET_DATA_TOMBSTONED (TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54670
I20240503 02:13:21.111172  9976 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f44b2b53de84a87a8757c0d00bfed37 with delete_type TABLET_DATA_TOMBSTONED (TS b85e85e631ab4e04a86bfe5a9f0aa512 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56956
W20240503 02:13:21.122061  8035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.1:37681, user_credentials={real_user=slave}} blocked reactor thread for 123442us
I20240503 02:13:21.122443  8360 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:13:21.123140  7817 catalog_manager.cc:4797] TS b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): delete failed for tablet 0f44b2b53de84a87a8757c0d00bfed37 because the tablet was not found. No further retry: Not found: Tablet not found: 0f44b2b53de84a87a8757c0d00bfed37
I20240503 02:13:21.123342 11460 tablet_replica.cc:330] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: stopping tablet replica
I20240503 02:13:21.123908  8232 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.124842  7967 raft_consensus.cc:858] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }.
  Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }: 
  New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
W20240503 02:13:21.124974  7967 raft_consensus.cc:1565] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 10' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }.
  Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }: 
  New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:21.125403  7967 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.126947  8229 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.123868 11460 raft_consensus.cc:2229] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:21.130937 11460 pending_rounds.cc:62] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Trying to abort 1 pending ops.
I20240503 02:13:21.131119 11460 pending_rounds.cc:68] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023421032342196224 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "5ea56405515a40f785726bd58ec3d26a" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } } }
I20240503 02:13:21.132175 11324 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:21.135062 11460 raft_consensus.cc:2894] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 02:13:21.135247 11460 raft_consensus.cc:2866] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:21.135416 11460 raft_consensus.cc:2258] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:21.136745  7971 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.142408 11460 ts_tablet_manager.cc:1875] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:21.146384 11083 ts_tablet_manager.cc:902] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512: Initiating tablet copy from peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:21.150182  7967 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.156039 11198 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:21.157716  8361 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:21.116947 10260 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.166271 11288 tablet_bootstrap.cc:492] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:21.180433 11460 ts_tablet_manager.cc:1888] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:21.181506 11460 log.cc:1196] T 5ea56405515a40f785726bd58ec3d26a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/5ea56405515a40f785726bd58ec3d26a
I20240503 02:13:21.182298 11288 tablet_bootstrap.cc:492] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:13:21.182806 11288 ts_tablet_manager.cc:1367] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 0.962s	user 0.055s	sys 0.003s
I20240503 02:13:21.183712 11288 raft_consensus.cc:348] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:21.183996 11288 raft_consensus.cc:729] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:13:21.184248 11288 consensus_queue.cc:260] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:21.186923  7816 catalog_manager.cc:4830] TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): tablet 5ea56405515a40f785726bd58ec3d26a (table rebalance_test_table_4 [id=c7d72eaa895849fcb02235c1b6e8d08f]) successfully deleted
I20240503 02:13:21.187417 11386 tablet_copy_client.cc:308] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:21.191637 11288 ts_tablet_manager.cc:1398] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:13:21.192685  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-d3d4b912095244de816d3089a31b8649 received from {username='slave'} at 127.0.145.7:43609
I20240503 02:13:21.192888  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-d3d4b912095244de816d3089a31b8649 on tablet d3d4b912095244de816d3089a31b8649 with peer d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:21.208195  7822 catalog_manager.cc:5484] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 4 to 5, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.212106  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet 7e966caa651e4e42890be5d79f851a1f from peer ca05c9a8bee148dab8a2a982b8fa2ea4 ({username='slave'} at 127.0.145.8:44853)
I20240503 02:13:21.212328  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet 7e966caa651e4e42890be5d79f851a1f from peer ca05c9a8bee148dab8a2a982b8fa2ea4 at {username='slave'} at 127.0.145.8:44853: session id = ca05c9a8bee148dab8a2a982b8fa2ea4-7e966caa651e4e42890be5d79f851a1f
I20240503 02:13:21.213613  7950 tablet_service.cc:1514] Processing DeleteTablet for tablet f14607425da84bb4b162a5789daa844c with delete_type TABLET_DATA_TOMBSTONED (TS d986fcfb926e47baa7d2d5b99f697334 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34494
I20240503 02:13:21.215456  8252 tablet_copy_source_session.cc:205] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:21.223654  7821 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 9 to 10, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.228710 11483 tablet_replica.cc:330] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334: stopping tablet replica
I20240503 02:13:21.230764 11483 raft_consensus.cc:2229] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:21.234220 11386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e966caa651e4e42890be5d79f851a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.234897 11483 raft_consensus.cc:2258] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:21.241847 11483 ts_tablet_manager.cc:1875] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:21.243731  8101 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:21.245721  8099 consensus_queue.cc:237] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:21.248550  9864 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:21.249579 11447 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:21.251523 11447 raft_consensus.cc:1050] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER 03f99420df704238ab300ad6e58d878c to VOTER
I20240503 02:13:21.252481 11447 consensus_queue.cc:237] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:21.253614 10319 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.257838  9862 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:21.258080  8035 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72e2bb75da654e8ab45bba8ac077f28c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.258719  9864 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
W20240503 02:13:21.259474  8035 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.260551  8360 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:21.261193 11136 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:21.263314  7967 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:21.264145  9862 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:21.264045 11324 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:21.264219 11447 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:21.264951 11324 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:21.267297 11324 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.269295  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f44b2b53de84a87a8757c0d00bfed37 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:21.270927 11257 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:21.271780  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aeaee03abc644807b8278cfb78e4f02b with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:21.272794 10128 raft_consensus.cc:1203] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:21.273042 11257 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:13:21.272704  9863 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.275676 11257 ts_tablet_manager.cc:1367] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 1.064s	user 0.055s	sys 0.005s
I20240503 02:13:21.274133  7971 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.271580  9864 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.277535 11257 raft_consensus.cc:348] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:21.279844 11257 raft_consensus.cc:729] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:13:21.280472 11257 consensus_queue.cc:260] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:21.283082 11257 ts_tablet_manager.cc:1398] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240503 02:13:21.284571  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-7e4acc8b5ceb4cd5b0f3708ff4f19298 received from {username='slave'} at 127.0.145.7:43609
I20240503 02:13:21.284740  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-7e4acc8b5ceb4cd5b0f3708ff4f19298 on tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 with peer d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:21.285383 11483 ts_tablet_manager.cc:1888] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 02:13:21.285604 11483 log.cc:1196] T f14607425da84bb4b162a5789daa844c P d986fcfb926e47baa7d2d5b99f697334: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/f14607425da84bb4b162a5789daa844c
I20240503 02:13:21.286967  7814 catalog_manager.cc:4830] TS d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): tablet f14607425da84bb4b162a5789daa844c (table rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]) successfully deleted
I20240503 02:13:21.288141  7828 catalog_manager.cc:5484] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 5 to 6, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.288710 11119 tablet_bootstrap.cc:492] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:21.289295 11119 tablet_bootstrap.cc:492] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:13:21.289695 11119 ts_tablet_manager.cc:1367] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 0.359s	user 0.049s	sys 0.007s
I20240503 02:13:21.308068 11447 raft_consensus.cc:1013] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: attempt to promote peer 03f99420df704238ab300ad6e58d878c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 02:13:21.308430 11134 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.312723  8361 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:21.313529 11447 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:21.315414  8361 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.319319 11119 raft_consensus.cc:348] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:21.319811 11119 raft_consensus.cc:729] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:13:21.320103 11119 consensus_queue.cc:260] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:21.320874 11119 ts_tablet_manager.cc:1398] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.031s	user 0.000s	sys 0.000s
I20240503 02:13:21.321735  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-aeaee03abc644807b8278cfb78e4f02b received from {username='slave'} at 127.0.145.7:59845
I20240503 02:13:21.321918  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-aeaee03abc644807b8278cfb78e4f02b on tablet aeaee03abc644807b8278cfb78e4f02b with peer d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:21.327585  9995 raft_consensus.cc:1203] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:21.329888  7823 catalog_manager.cc:5484] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 6 to 7, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.336879  7829 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c reported cstate change: config changed from index 10 to 11, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.340111  8211 tablet_service.cc:1514] Processing DeleteTablet for tablet aeaee03abc644807b8278cfb78e4f02b with delete_type TABLET_DATA_TOMBSTONED (TS 2942efc0ecb448d2ba2f6bca20fbb29c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35426
I20240503 02:13:21.343271 11494 tablet_replica.cc:330] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c: stopping tablet replica
I20240503 02:13:21.343801 11494 raft_consensus.cc:2229] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:21.344238 11494 raft_consensus.cc:2258] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:21.344692 11083 tablet_copy_client.cc:308] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:37681
I20240503 02:13:21.345420  7991 tablet_copy_service.cc:140] P d986fcfb926e47baa7d2d5b99f697334: Received BeginTabletCopySession request for tablet f231fe5dfd9e4c87b90bad75818a7060 from peer b85e85e631ab4e04a86bfe5a9f0aa512 ({username='slave'} at 127.0.145.6:46235)
I20240503 02:13:21.345626  7991 tablet_copy_service.cc:161] P d986fcfb926e47baa7d2d5b99f697334: Beginning new tablet copy session on tablet f231fe5dfd9e4c87b90bad75818a7060 from peer b85e85e631ab4e04a86bfe5a9f0aa512 at {username='slave'} at 127.0.145.6:46235: session id = b85e85e631ab4e04a86bfe5a9f0aa512-f231fe5dfd9e4c87b90bad75818a7060
I20240503 02:13:21.345682  7950 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f44b2b53de84a87a8757c0d00bfed37 with delete_type TABLET_DATA_TOMBSTONED (TS d986fcfb926e47baa7d2d5b99f697334 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:34494
I20240503 02:13:21.346010 11483 tablet_replica.cc:330] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334: stopping tablet replica
I20240503 02:13:21.346222 11483 raft_consensus.cc:2229] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:21.346877 11483 raft_consensus.cc:2258] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:21.347290  8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f44b2b53de84a87a8757c0d00bfed37 with delete_type TABLET_DATA_TOMBSTONED (TS 8d7c037d50c241b484ee843b06313a64 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:37080
I20240503 02:13:21.348270 11483 ts_tablet_manager.cc:1875] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:21.349073  7991 tablet_copy_source_session.cc:205] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:21.350574 11083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f231fe5dfd9e4c87b90bad75818a7060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.358062 11494 ts_tablet_manager.cc:1875] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:21.363539 11497 ts_tablet_manager.cc:902] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:21.387269 11083 tablet_copy_client.cc:783] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:21.387611 11083 tablet_copy_client.cc:647] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:21.388495 11496 tablet_replica.cc:330] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64: stopping tablet replica
I20240503 02:13:21.389004 11496 raft_consensus.cc:2229] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:21.389976 11496 raft_consensus.cc:2258] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:21.396943 10261 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:21.398191 11447 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:21.400008 10261 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.406502 11483 ts_tablet_manager.cc:1888] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 02:13:21.407892 11505 ts_tablet_manager.cc:902] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:21.408924 11386 tablet_copy_client.cc:783] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:21.409204 11386 tablet_copy_client.cc:647] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:21.409236 11494 ts_tablet_manager.cc:1888] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:13:21.409441 11494 log.cc:1196] T aeaee03abc644807b8278cfb78e4f02b P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/aeaee03abc644807b8278cfb78e4f02b
I20240503 02:13:21.411036  7814 catalog_manager.cc:4830] TS 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): tablet aeaee03abc644807b8278cfb78e4f02b (table rebalance_test_table_9 [id=5f6139f12f9b4ab18ce35a8831b39150]) successfully deleted
I20240503 02:13:21.422044  8362 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.430819 11483 log.cc:1196] T 0f44b2b53de84a87a8757c0d00bfed37 P d986fcfb926e47baa7d2d5b99f697334: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/0f44b2b53de84a87a8757c0d00bfed37
I20240503 02:13:21.431921  7814 catalog_manager.cc:4830] TS d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): tablet 0f44b2b53de84a87a8757c0d00bfed37 (table rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]) successfully deleted
I20240503 02:13:21.441092 11196 tablet_bootstrap.cc:492] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:21.442034 11196 tablet_bootstrap.cc:492] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:21.442427 11196 ts_tablet_manager.cc:1367] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.913s	user 0.063s	sys 0.002s
I20240503 02:13:21.443444 11196 raft_consensus.cc:348] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:21.443794 11196 raft_consensus.cc:729] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:21.445482 11227 raft_consensus.cc:1050] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 to VOTER
I20240503 02:13:21.444242 11196 consensus_queue.cc:260] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:21.447466 11196 ts_tablet_manager.cc:1398] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:13:21.448418 11505 tablet_copy_client.cc:308] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:37681
I20240503 02:13:21.451503  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-ceae1c7c45a34b24badbbe23cb9e0176 received from {username='slave'} at 127.0.145.5:37535
I20240503 02:13:21.451689  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session 03f99420df704238ab300ad6e58d878c-ceae1c7c45a34b24badbbe23cb9e0176 on tablet ceae1c7c45a34b24badbbe23cb9e0176 with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:21.453776  8099 consensus_queue.cc:237] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:21.455974 11083 tablet_copy_client.cc:523] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:21.457494  9864 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:21.458824 11324 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:21.459926  7971 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:21.462540  8035 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72e2bb75da654e8ab45bba8ac077f28c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.463047 11136 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:21.463716  9864 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.464110 10319 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.468472  7971 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.469944 11289 tablet_bootstrap.cc:492] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:21.470494 11289 tablet_bootstrap.cc:492] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap complete.
I20240503 02:13:21.470854 11289 ts_tablet_manager.cc:1367] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent bootstrapping tablet: real 0.378s	user 0.038s	sys 0.015s
I20240503 02:13:21.471771 11289 raft_consensus.cc:348] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:21.472045 11289 raft_consensus.cc:729] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Initialized, Role: LEARNER
I20240503 02:13:21.472286 11289 consensus_queue.cc:260] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:21.472867 11289 ts_tablet_manager.cc:1398] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:13:21.474922  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 72e2bb75da654e8ab45bba8ac077f28c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:21.475112  7821 catalog_manager.cc:5484] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c reported cstate change: config changed from index 6 to 7, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.479326 11083 tablet_bootstrap.cc:492] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap starting.
I20240503 02:13:21.484398  9864 raft_consensus.cc:1203] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:21.494745  8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 72e2bb75da654e8ab45bba8ac077f28c with delete_type TABLET_DATA_TOMBSTONED (TS 8d7c037d50c241b484ee843b06313a64 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37080
I20240503 02:13:21.497406  7991 tablet_copy_service.cc:140] P d986fcfb926e47baa7d2d5b99f697334: Received BeginTabletCopySession request for tablet 4f2cedba9f9a4e709623fceeffcfd095 from peer ca05c9a8bee148dab8a2a982b8fa2ea4 ({username='slave'} at 127.0.145.8:52797)
I20240503 02:13:21.497651  7991 tablet_copy_service.cc:161] P d986fcfb926e47baa7d2d5b99f697334: Beginning new tablet copy session on tablet 4f2cedba9f9a4e709623fceeffcfd095 from peer ca05c9a8bee148dab8a2a982b8fa2ea4 at {username='slave'} at 127.0.145.8:52797: session id = ca05c9a8bee148dab8a2a982b8fa2ea4-4f2cedba9f9a4e709623fceeffcfd095
I20240503 02:13:21.501195  7991 tablet_copy_source_session.cc:205] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:21.503026 11505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2cedba9f9a4e709623fceeffcfd095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.514331  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-3dbfc7f12fc142f98b1d1c026fd49f5a received from {username='slave'} at 127.0.145.8:38819
I20240503 02:13:21.516741 11505 tablet_copy_client.cc:783] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:21.517699 11505 tablet_copy_client.cc:647] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:21.518630  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-3dbfc7f12fc142f98b1d1c026fd49f5a on tablet 3dbfc7f12fc142f98b1d1c026fd49f5a with peer ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:21.523751 11163 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:21.524456 11163 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap complete.
I20240503 02:13:21.524765 11163 ts_tablet_manager.cc:1367] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent bootstrapping tablet: real 0.828s	user 0.054s	sys 0.006s
I20240503 02:13:21.525727 11163 raft_consensus.cc:348] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:21.526005 11163 raft_consensus.cc:729] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Initialized, Role: LEARNER
I20240503 02:13:21.531129 11505 tablet_copy_client.cc:523] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:21.544504 11496 ts_tablet_manager.cc:1875] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:21.551712 10259 raft_consensus.cc:1203] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:21.554777 11386 tablet_copy_client.cc:523] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:21.555379  7971 consensus_queue.cc:237] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:21.550127 11227 consensus_queue.cc:237] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:21.562134  7967 consensus_queue.cc:237] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:21.564272 11505 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap starting.
I20240503 02:13:21.565150  8099 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:21.565879 11401 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:21.569033  8231 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:21.569489 10128 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:21.569768 11401 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:21.571898 10659 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:21.573311 11401 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:21.579012  8231 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:21.526238 11163 consensus_queue.cc:260] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:21.584937 11163 ts_tablet_manager.cc:1398] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent starting tablet: real 0.060s	user 0.004s	sys 0.000s
I20240503 02:13:21.587255  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4f2cedba9f9a4e709623fceeffcfd095 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:21.587028 10128 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:21.588641  7829 catalog_manager.cc:5484] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 6 to 7, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) evicted. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:21.594568  8099 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:21.596683  8362 consensus_queue.cc:237] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:21.599061 11497 tablet_copy_client.cc:308] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:21.599815  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet c23b06764ce54acb8388de94aea4cd4b from peer b85e85e631ab4e04a86bfe5a9f0aa512 ({username='slave'} at 127.0.145.6:49885)
I20240503 02:13:21.600000  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet c23b06764ce54acb8388de94aea4cd4b from peer b85e85e631ab4e04a86bfe5a9f0aa512 at {username='slave'} at 127.0.145.6:49885: session id = b85e85e631ab4e04a86bfe5a9f0aa512-c23b06764ce54acb8388de94aea4cd4b
I20240503 02:13:21.603547  8382 tablet_copy_source_session.cc:205] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:21.604156  7967 consensus_queue.cc:237] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:21.606762 11497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23b06764ce54acb8388de94aea4cd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.609290 10128 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:21.609265  8100 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:21.609872 10659 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:21.610474 11400 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:21.613678 11400 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:21.617523  8100 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:21.624151  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4f2cedba9f9a4e709623fceeffcfd095 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:21.624881  8231 raft_consensus.cc:1261] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:21.625685 11514 consensus_queue.cc:1035] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:21.624645 10128 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:21.626315  7829 catalog_manager.cc:5484] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
W20240503 02:13:21.631170  8295 consensus_peers.cc:485] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ba1f505010b403c91d3870cee19952a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.631179  7967 raft_consensus.cc:1261] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:21.631696  7906 consensus_peers.cc:485] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c45c5a14b85447c78fe2a27be4392c88. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.631731  8361 raft_consensus.cc:1261] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:21.632588 11514 consensus_queue.cc:1035] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:21.633222  8100 raft_consensus.cc:1261] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:21.633853 11401 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:21.634294 11400 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:21.634474  8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f2cedba9f9a4e709623fceeffcfd095 with delete_type TABLET_DATA_TOMBSTONED (TS 2942efc0ecb448d2ba2f6bca20fbb29c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35426
I20240503 02:13:21.634938 11494 tablet_replica.cc:330] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c: stopping tablet replica
I20240503 02:13:21.635193 11494 raft_consensus.cc:2229] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:21.635607 11494 raft_consensus.cc:2258] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:21.636790 11494 ts_tablet_manager.cc:1875] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:21.638303  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-32e527fa9b2f441cb5668435681b4ff6 received from {username='slave'} at 127.0.145.6:56443
I20240503 02:13:21.638427  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-32e527fa9b2f441cb5668435681b4ff6 on tablet 32e527fa9b2f441cb5668435681b4ff6 with peer b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:21.647305 11494 ts_tablet_manager.cc:1888] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 02:13:21.649242 11494 log.cc:1196] T 4f2cedba9f9a4e709623fceeffcfd095 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/4f2cedba9f9a4e709623fceeffcfd095
I20240503 02:13:21.650172  7814 catalog_manager.cc:4830] TS 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): tablet 4f2cedba9f9a4e709623fceeffcfd095 (table rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]) successfully deleted
I20240503 02:13:21.650967 11514 raft_consensus.cc:2932] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.658572  8231 raft_consensus.cc:2932] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.660688  7821 catalog_manager.cc:5484] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:21.667613 11401 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:21.670847  7967 raft_consensus.cc:2932] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.676915  8100 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:21.678190  8362 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:21.680081  7823 catalog_manager.cc:5484] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:21.699431 10259 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:21.701442 10995 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:21.704712 11386 tablet_bootstrap.cc:492] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap starting.
I20240503 02:13:21.710777  8100 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:21.712172 11227 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:21.730546 10241 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f2cedba9f9a4e709623fceeffcfd095 with delete_type TABLET_DATA_TOMBSTONED (TS ca05c9a8bee148dab8a2a982b8fa2ea4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46856
I20240503 02:13:21.736182  8231 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:21.737069 11227 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:13:21.766745 11505 log.cc:924] Time spent T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20240503 02:13:21.772133 11496 ts_tablet_manager.cc:1888] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:13:21.772358 11496 log.cc:1196] T 0f44b2b53de84a87a8757c0d00bfed37 P 8d7c037d50c241b484ee843b06313a64: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/wals/0f44b2b53de84a87a8757c0d00bfed37
I20240503 02:13:21.772920 11516 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.777922  7817 catalog_manager.cc:4830] TS 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): tablet 0f44b2b53de84a87a8757c0d00bfed37 (table rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]) successfully deleted
W20240503 02:13:21.780575  7816 catalog_manager.cc:4810] TS ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): delete failed for tablet 4f2cedba9f9a4e709623fceeffcfd095 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f2cedba9f9a4e709623fceeffcfd095 already in progress: copying tablet
I20240503 02:13:21.781560  8231 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.782585 10259 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.790511 11497 tablet_copy_client.cc:783] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:21.790936 11497 tablet_copy_client.cc:647] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:21.794377 10128 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:21.794981  8099 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.797766  7829 catalog_manager.cc:5484] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:21.801584 11134 consensus_queue.cc:1037] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:21.805768 10128 raft_consensus.cc:858] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
W20240503 02:13:21.805930 10128 raft_consensus.cc:1565] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:21.773470 11496 tablet_replica.cc:330] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: stopping tablet replica
I20240503 02:13:21.809911 11496 raft_consensus.cc:2229] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:21.810200 11496 raft_consensus.cc:2258] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:21.811865 11496 ts_tablet_manager.cc:1875] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:21.828039 11317 tablet_bootstrap.cc:492] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:21.832845 11317 tablet_bootstrap.cc:492] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap complete.
I20240503 02:13:21.836212  8099 consensus_queue.cc:237] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:21.837182 11317 ts_tablet_manager.cc:1367] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent bootstrapping tablet: real 1.249s	user 0.051s	sys 0.015s
W20240503 02:13:21.848515  8038 consensus_peers.cc:485] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45fe6ec58eed40f1898d4a2f5727cfa1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.849057  7971 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:21.850147  8231 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:21.850064 11317 raft_consensus.cc:348] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:21.850107 10319 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:21.853945 11134 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:21.850320 11317 raft_consensus.cc:729] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Initialized, Role: LEARNER
I20240503 02:13:21.855209 11317 consensus_queue.cc:260] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:21.857529 11497 tablet_copy_client.cc:523] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:21.858321 11317 ts_tablet_manager.cc:1398] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent starting tablet: real 0.015s	user 0.002s	sys 0.000s
I20240503 02:13:21.860291 11518 raft_consensus.cc:1050] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f to VOTER
I20240503 02:13:21.861256 11518 consensus_queue.cc:237] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:21.863732  7991 tablet_copy_service.cc:342] P d986fcfb926e47baa7d2d5b99f697334: Request end of tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-cab70b62bdc747a9b653bffde680f37e received from {username='slave'} at 127.0.145.6:46235
I20240503 02:13:21.863922  7991 tablet_copy_service.cc:433] P d986fcfb926e47baa7d2d5b99f697334: ending tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-cab70b62bdc747a9b653bffde680f37e on tablet cab70b62bdc747a9b653bffde680f37e with peer b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:21.869411 11227 raft_consensus.cc:1050] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 to VOTER
I20240503 02:13:21.870328 11227 consensus_queue.cc:237] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:21.876288 11496 ts_tablet_manager.cc:1888] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:13:21.877285 10128 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.882944 11496 log.cc:1196] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/wals/72e2bb75da654e8ab45bba8ac077f28c
I20240503 02:13:21.883981  7817 catalog_manager.cc:4830] TS 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): tablet 72e2bb75da654e8ab45bba8ac077f28c (table rebalance_test_table_8 [id=8da59f26450d4d25a86bc61be2943c37]) successfully deleted
I20240503 02:13:21.884070 10127 raft_consensus.cc:1203] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:21.898479 10319 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:21.902874 11349 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:21.903450 11349 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap complete.
I20240503 02:13:21.903983 11349 ts_tablet_manager.cc:1367] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent bootstrapping tablet: real 0.970s	user 0.067s	sys 0.000s
I20240503 02:13:21.905087 11349 raft_consensus.cc:348] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:21.904954  7971 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:21.907447 11349 raft_consensus.cc:729] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Initialized, Role: LEARNER
I20240503 02:13:21.907930 11349 consensus_queue.cc:260] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:21.907837  8231 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:21.912462  7821 catalog_manager.cc:5484] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index -1 to 4, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:21.918301  8099 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:21.919162  7971 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:21.921043 11514 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:21.921813 11514 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:21.926165 11505 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:21.917061  9995 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:21.931540 11505 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap complete.
I20240503 02:13:21.932983 11514 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:21.933301 11505 ts_tablet_manager.cc:1367] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent bootstrapping tablet: real 0.369s	user 0.052s	sys 0.012s
I20240503 02:13:21.933357 11349 ts_tablet_manager.cc:1398] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent starting tablet: real 0.029s	user 0.002s	sys 0.000s
I20240503 02:13:21.934101  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-4af5884849ec4ac3891ebf0f08ee2cff received from {username='slave'} at 127.0.145.6:56611
I20240503 02:13:21.934258  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-4af5884849ec4ac3891ebf0f08ee2cff on tablet 4af5884849ec4ac3891ebf0f08ee2cff with peer b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:21.934352 11505 raft_consensus.cc:348] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:21.934701 11505 raft_consensus.cc:729] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Initialized, Role: LEARNER
I20240503 02:13:21.936347 11505 consensus_queue.cc:260] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:21.940186 11505 ts_tablet_manager.cc:1398] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240503 02:13:21.941725  7991 tablet_copy_service.cc:342] P d986fcfb926e47baa7d2d5b99f697334: Request end of tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-4f2cedba9f9a4e709623fceeffcfd095 received from {username='slave'} at 127.0.145.8:52797
I20240503 02:13:21.941896  7991 tablet_copy_service.cc:433] P d986fcfb926e47baa7d2d5b99f697334: ending tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-4f2cedba9f9a4e709623fceeffcfd095 on tablet 4f2cedba9f9a4e709623fceeffcfd095 with peer ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:21.942198 11497 tablet_bootstrap.cc:492] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap starting.
I20240503 02:13:21.944479 10241 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f2cedba9f9a4e709623fceeffcfd095 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 6 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:46856
I20240503 02:13:21.944866 11549 tablet_replica.cc:330] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: stopping tablet replica
I20240503 02:13:21.945442 11549 raft_consensus.cc:2229] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 02:13:21.945806 11549 raft_consensus.cc:2258] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 02:13:21.947489 11549 ts_tablet_manager.cc:1875] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:21.951793 11578 ts_tablet_manager.cc:902] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:21.953675 11516 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:21.955004  8100 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:21.961570  7821 catalog_manager.cc:5484] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:21.962868  9995 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:21.969538  8361 consensus_queue.cc:237] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:21.973424  9996 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:21.974717 11227 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:21.976982  9996 raft_consensus.cc:1203] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:21.977640  8099 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:21.978513 11227 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:21.980748 10319 raft_consensus.cc:1050] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f to VOTER
I20240503 02:13:21.981525 10319 consensus_queue.cc:237] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:21.982190 11516 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:21.984160  9996 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:21.992538 10127 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:21.993659 11517 raft_consensus.cc:1050] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 to VOTER
I20240503 02:13:21.994635 11517 consensus_queue.cc:237] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:21.995364  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80b4e6aa20f64473be8bcbf47baf66e4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:21.996265  9864 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:21.997058 11324 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:21.998989  8231 raft_consensus.cc:1261] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:21.999234  7967 raft_consensus.cc:1261] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:21.999466 11549 ts_tablet_manager.cc:1888] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:13:21.999672 11549 log.cc:1196] T 4f2cedba9f9a4e709623fceeffcfd095 P ca05c9a8bee148dab8a2a982b8fa2ea4: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-7/wal/wals/4f2cedba9f9a4e709623fceeffcfd095
I20240503 02:13:21.999737 11514 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.000651  7816 catalog_manager.cc:4830] TS ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): tablet 4f2cedba9f9a4e709623fceeffcfd095 (table rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]) successfully deleted
I20240503 02:13:22.001960  8361 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:22.002763 10319 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:22.003326 11514 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.005190 11324 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:21.998399  7829 catalog_manager.cc:5484] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:22.010290  8361 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.011297  7950 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b4e6aa20f64473be8bcbf47baf66e4 with delete_type TABLET_DATA_TOMBSTONED (TS d986fcfb926e47baa7d2d5b99f697334 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34494
I20240503 02:13:22.020399  9864 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.032785  7828 catalog_manager.cc:5484] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 7 to 8, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.043618 10128 raft_consensus.cc:1261] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.045154 11516 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.046537  8099 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.052223 11254 raft_consensus.cc:1050] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER 03f99420df704238ab300ad6e58d878c to VOTER
I20240503 02:13:22.053102 11254 consensus_queue.cc:237] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:22.061484  9863 raft_consensus.cc:1261] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.063072  7967 raft_consensus.cc:1261] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.063743 11198 consensus_queue.cc:1035] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.066768 11198 consensus_queue.cc:1035] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.070253  8100 raft_consensus.cc:1261] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.071035 11198 consensus_queue.cc:1035] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.081863 10259 raft_consensus.cc:1261] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.082070  8101 raft_consensus.cc:1261] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.082840 11514 consensus_queue.cc:1035] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.083436 11514 consensus_queue.cc:1035] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.085847  8230 raft_consensus.cc:1261] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.087059 11518 consensus_queue.cc:1035] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.091528  9995 raft_consensus.cc:1203] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:22.112161 11518 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.113075  8231 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.113762 11607 ts_tablet_manager.cc:902] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:22.117167  7971 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.119144  7821 catalog_manager.cc:5484] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:22.134356 11254 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:22.136611 11514 raft_consensus.cc:2932] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:22.142809  8101 raft_consensus.cc:2932] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:22.144284  7828 catalog_manager.cc:5484] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:22.145282 10259 raft_consensus.cc:2932] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:22.156955  7829 catalog_manager.cc:5484] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 4 to 5, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:22.157732 10124 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.162110  7967 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:22.167160  8231 raft_consensus.cc:2932] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:22.170284  8361 consensus_queue.cc:237] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:22.173059  8099 raft_consensus.cc:1261] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:22.175519 11597 tablet_replica.cc:330] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334: stopping tablet replica
I20240503 02:13:22.175879 11597 raft_consensus.cc:2229] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:22.176244 11597 pending_rounds.cc:62] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334: Trying to abort 1 pending ops.
I20240503 02:13:22.176442 11597 pending_rounds.cc:68] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023421038058590208 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "80b4e6aa20f64473be8bcbf47baf66e4" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } } }
I20240503 02:13:22.176970  8361 consensus_queue.cc:237] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:22.177171 11597 raft_consensus.cc:2894] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 02:13:22.177337 11597 raft_consensus.cc:2866] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:22.177459 11597 raft_consensus.cc:2258] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.183501  8101 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:22.187774 11518 consensus_queue.cc:1035] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:22.189358 10126 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:22.190222 10319 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:22.191192 11517 raft_consensus.cc:2932] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:22.192031 10126 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.169649  9864 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:22.195670  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3dbfc7f12fc142f98b1d1c026fd49f5a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:22.197553 11505 ts_tablet_manager.cc:902] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:22.197505  8101 raft_consensus.cc:2932] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:22.197973  7822 catalog_manager.cc:5484] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 5 to 6, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:22.201045  8232 raft_consensus.cc:1261] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:22.201936 11517 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:22.202404  8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dbfc7f12fc142f98b1d1c026fd49f5a with delete_type TABLET_DATA_TOMBSTONED (TS 2942efc0ecb448d2ba2f6bca20fbb29c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35426
I20240503 02:13:22.205459 11516 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.209091  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3d4b912095244de816d3089a31b8649 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:22.209790 10260 raft_consensus.cc:1261] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:22.210122  8230 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.210903 10125 raft_consensus.cc:1261] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:22.213908 10995 consensus_queue.cc:1035] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:22.216898 11227 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:22.218945 10124 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.223279  7822 catalog_manager.cc:5484] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:22.223466 10259 raft_consensus.cc:2932] T 3dbfc7f12fc142f98b1d1c026fd49f5a P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:22.230232 11607 tablet_copy_client.cc:308] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:22.223186 11597 ts_tablet_manager.cc:1875] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:22.232761  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet 5ba1f505010b403c91d3870cee19952a from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:43609)
I20240503 02:13:22.232944  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet 5ba1f505010b403c91d3870cee19952a from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:43609: session id = d826d4dae2aa4e05b2f64c9b17a1126f-5ba1f505010b403c91d3870cee19952a
I20240503 02:13:22.229754 11505 tablet_copy_client.cc:308] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:37681
I20240503 02:13:22.236338  8382 tablet_copy_source_session.cc:205] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.236630  7991 tablet_copy_service.cc:140] P d986fcfb926e47baa7d2d5b99f697334: Received BeginTabletCopySession request for tablet c45c5a14b85447c78fe2a27be4392c88 from peer ca05c9a8bee148dab8a2a982b8fa2ea4 ({username='slave'} at 127.0.145.8:52797)
I20240503 02:13:22.236832  7991 tablet_copy_service.cc:161] P d986fcfb926e47baa7d2d5b99f697334: Beginning new tablet copy session on tablet c45c5a14b85447c78fe2a27be4392c88 from peer ca05c9a8bee148dab8a2a982b8fa2ea4 at {username='slave'} at 127.0.145.8:52797: session id = ca05c9a8bee148dab8a2a982b8fa2ea4-c45c5a14b85447c78fe2a27be4392c88
I20240503 02:13:22.237149  8232 consensus_queue.cc:237] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:22.238343 11607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba1f505010b403c91d3870cee19952a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.239743  7991 tablet_copy_source_session.cc:205] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.240514  7950 tablet_service.cc:1514] Processing DeleteTablet for tablet d3d4b912095244de816d3089a31b8649 with delete_type TABLET_DATA_TOMBSTONED (TS d986fcfb926e47baa7d2d5b99f697334 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34494
I20240503 02:13:22.241420 11505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45c5a14b85447c78fe2a27be4392c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.242434 11627 tablet_replica.cc:330] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c: stopping tablet replica
I20240503 02:13:22.242789 11627 raft_consensus.cc:2229] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:22.246033 11627 raft_consensus.cc:2258] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.247069  8099 raft_consensus.cc:1261] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:22.247433 11627 ts_tablet_manager.cc:1875] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:22.247756 11618 consensus_queue.cc:1035] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:22.250566 11254 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:22.254426 11578 tablet_copy_client.cc:308] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:22.255331  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 72e2bb75da654e8ab45bba8ac077f28c from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:59845)
I20240503 02:13:22.255487  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 72e2bb75da654e8ab45bba8ac077f28c from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:59845: session id = d826d4dae2aa4e05b2f64c9b17a1126f-72e2bb75da654e8ab45bba8ac077f28c
I20240503 02:13:22.256263  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ceae1c7c45a34b24badbbe23cb9e0176 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:22.257122  7824 catalog_manager.cc:5484] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:22.257860  8100 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:22.258534  8122 tablet_copy_source_session.cc:205] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.261919 11578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e2bb75da654e8ab45bba8ac077f28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.265055  7950 tablet_service.cc:1514] Processing DeleteTablet for tablet ceae1c7c45a34b24badbbe23cb9e0176 with delete_type TABLET_DATA_TOMBSTONED (TS d986fcfb926e47baa7d2d5b99f697334 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34494
I20240503 02:13:22.273929 11627 ts_tablet_manager.cc:1888] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:22.274302 11627 log.cc:1196] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/3dbfc7f12fc142f98b1d1c026fd49f5a
I20240503 02:13:22.275400  7814 catalog_manager.cc:4830] TS 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): tablet 3dbfc7f12fc142f98b1d1c026fd49f5a (table rebalance_test_table_1 [id=d4270f8900eb45459ee1e49c30ff6cff]) successfully deleted
I20240503 02:13:22.277626 11083 tablet_bootstrap.cc:492] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.278106 11083 tablet_bootstrap.cc:492] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap complete.
I20240503 02:13:22.279686 11083 ts_tablet_manager.cc:1367] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent bootstrapping tablet: real 0.806s	user 0.053s	sys 0.003s
I20240503 02:13:22.280555 11083 raft_consensus.cc:348] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:22.280813 11083 raft_consensus.cc:729] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Initialized, Role: LEARNER
I20240503 02:13:22.293124 11505 tablet_copy_client.cc:783] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:22.293493 11505 tablet_copy_client.cc:647] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:22.296025 11597 ts_tablet_manager.cc:1888] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:22.296286 11597 log.cc:1196] T 80b4e6aa20f64473be8bcbf47baf66e4 P d986fcfb926e47baa7d2d5b99f697334: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/80b4e6aa20f64473be8bcbf47baf66e4
I20240503 02:13:22.297168 11597 tablet_replica.cc:330] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334: stopping tablet replica
I20240503 02:13:22.297439 11597 raft_consensus.cc:2229] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:22.297793 11597 raft_consensus.cc:2258] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.298385  7814 catalog_manager.cc:4830] TS d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): tablet 80b4e6aa20f64473be8bcbf47baf66e4 (table rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]) successfully deleted
I20240503 02:13:22.305387 11505 tablet_copy_client.cc:523] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:22.281018 11083 consensus_queue.cc:260] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:22.307523 11083 ts_tablet_manager.cc:1398] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent starting tablet: real 0.028s	user 0.001s	sys 0.001s
I20240503 02:13:22.310590  9863 raft_consensus.cc:1261] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:22.312041 11618 consensus_queue.cc:1035] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:22.324887  7991 tablet_copy_service.cc:342] P d986fcfb926e47baa7d2d5b99f697334: Request end of tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-f231fe5dfd9e4c87b90bad75818a7060 received from {username='slave'} at 127.0.145.6:46235
I20240503 02:13:22.325065  7991 tablet_copy_service.cc:433] P d986fcfb926e47baa7d2d5b99f697334: ending tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-f231fe5dfd9e4c87b90bad75818a7060 on tablet f231fe5dfd9e4c87b90bad75818a7060 with peer b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:22.324929 11597 ts_tablet_manager.cc:1875] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:22.328799  9864 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:22.335974 11505 tablet_bootstrap.cc:492] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap starting.
I20240503 02:13:22.346370 11083 ts_tablet_manager.cc:902] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:22.363338 11578 tablet_copy_client.cc:783] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:22.366425 11607 tablet_copy_client.cc:783] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:22.367007 11578 tablet_copy_client.cc:647] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:22.367244 11607 tablet_copy_client.cc:647] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:22.369474 11607 tablet_copy_client.cc:523] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:22.372601 11578 tablet_copy_client.cc:523] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:22.376262 11578 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
I20240503 02:13:22.375020 11597 ts_tablet_manager.cc:1888] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:22.378916 11597 log.cc:1196] T d3d4b912095244de816d3089a31b8649 P d986fcfb926e47baa7d2d5b99f697334: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/d3d4b912095244de816d3089a31b8649
I20240503 02:13:22.379812 11607 tablet_bootstrap.cc:492] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
I20240503 02:13:22.385188 11597 tablet_replica.cc:330] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334: stopping tablet replica
I20240503 02:13:22.385426 11597 raft_consensus.cc:2229] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:22.385672 11597 raft_consensus.cc:2258] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.386709  7814 catalog_manager.cc:4830] TS d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): tablet d3d4b912095244de816d3089a31b8649 (table rebalance_test_table_4 [id=c7d72eaa895849fcb02235c1b6e8d08f]) successfully deleted
I20240503 02:13:22.387259  9995 raft_consensus.cc:1203] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:22.388911 11597 ts_tablet_manager.cc:1875] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:22.408795 11386 tablet_bootstrap.cc:492] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.416301 11386 tablet_bootstrap.cc:492] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap complete.
I20240503 02:13:22.417906 11386 ts_tablet_manager.cc:1367] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent bootstrapping tablet: real 0.716s	user 0.048s	sys 0.023s
I20240503 02:13:22.422955 11386 raft_consensus.cc:348] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:22.429304 11386 raft_consensus.cc:729] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Initialized, Role: LEARNER
I20240503 02:13:22.429770 11386 consensus_queue.cc:260] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:22.430014 11597 ts_tablet_manager.cc:1888] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:22.431579 11386 ts_tablet_manager.cc:1398] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.000s
I20240503 02:13:22.432384  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-7e966caa651e4e42890be5d79f851a1f received from {username='slave'} at 127.0.145.8:44853
I20240503 02:13:22.432538  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-7e966caa651e4e42890be5d79f851a1f on tablet 7e966caa651e4e42890be5d79f851a1f with peer ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:22.435338 11597 log.cc:1196] T ceae1c7c45a34b24badbbe23cb9e0176 P d986fcfb926e47baa7d2d5b99f697334: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/ceae1c7c45a34b24badbbe23cb9e0176
I20240503 02:13:22.436300  7814 catalog_manager.cc:4830] TS d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): tablet ceae1c7c45a34b24badbbe23cb9e0176 (table rebalance_test_table_4 [id=c7d72eaa895849fcb02235c1b6e8d08f]) successfully deleted
I20240503 02:13:22.463068 11401 raft_consensus.cc:1050] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334: attempting to promote NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 to VOTER
I20240503 02:13:22.463385 11516 raft_consensus.cc:1050] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f to VOTER
I20240503 02:13:22.464165 11401 consensus_queue.cc:237] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:22.464208 11516 consensus_queue.cc:237] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:22.471097 10124 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.471417  7967 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.471724  8100 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.472617 11514 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.473277 11514 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.473750 11514 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.506106  9995 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.507582 10659 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.511612  8100 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.512884 11400 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.516453  8361 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.517999 10659 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.518805 10659 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.523592  8100 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.525897  9995 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.532925 11618 raft_consensus.cc:1050] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 to VOTER
I20240503 02:13:22.532840  7823 catalog_manager.cc:5484] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.533934 11618 consensus_queue.cc:237] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:22.547768  7971 consensus_queue.cc:237] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:22.551648  9996 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:22.552106  8100 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:22.552544 11400 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:22.555500 11518 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.557381 11400 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:22.559871 10124 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.561357 11400 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.562769  8100 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.564261  8101 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.569213  7825 catalog_manager.cc:5484] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.572324 11578 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.572804 11578 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:13:22.573098 11578 ts_tablet_manager.cc:1367] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 0.197s	user 0.046s	sys 0.006s
I20240503 02:13:22.573958 11578 raft_consensus.cc:348] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:22.574240 11578 raft_consensus.cc:729] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:13:22.575084  8362 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.576655 11254 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.580577  8360 consensus_queue.cc:237] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:22.583936  7830 catalog_manager.cc:5484] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 5 to 6, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.594425  8101 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:22.594748 10124 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:22.595391 11514 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:22.595875 11514 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:22.597136  8341 tablet_service.cc:1514] Processing DeleteTablet for tablet cab70b62bdc747a9b653bffde680f37e with delete_type TABLET_DATA_TOMBSTONED (TS 8d7c037d50c241b484ee843b06313a64 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37080
I20240503 02:13:22.599220 11514 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.574474 11578 consensus_queue.cc:260] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:22.601987 11578 ts_tablet_manager.cc:1398] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.029s	user 0.004s	sys 0.000s
I20240503 02:13:22.602828  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-72e2bb75da654e8ab45bba8ac077f28c received from {username='slave'} at 127.0.145.7:59845
I20240503 02:13:22.602981  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-72e2bb75da654e8ab45bba8ac077f28c on tablet 72e2bb75da654e8ab45bba8ac077f28c with peer d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:22.604058 10124 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
W20240503 02:13:22.605729 11676 log.cc:924] Time spent T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20240503 02:13:22.606215  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:22.628818  8100 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:22.637856  9995 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.638540 11254 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.653013  8101 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.653671 11254 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.608775  7828 catalog_manager.cc:5484] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:22.657783  8361 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.663467 11607 tablet_bootstrap.cc:492] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.663991 11607 tablet_bootstrap.cc:492] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:13:22.668521 10124 raft_consensus.cc:1203] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:13:22.677189 11607 ts_tablet_manager.cc:1367] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 0.285s	user 0.043s	sys 0.007s
I20240503 02:13:22.677969 11254 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.681085  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cab70b62bdc747a9b653bffde680f37e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:22.685010 11607 raft_consensus.cc:348] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:22.685314 11607 raft_consensus.cc:729] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:13:22.684692  8099 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.686553  7821 catalog_manager.cc:5484] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:22.691191  8360 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.691763 11607 consensus_queue.cc:260] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:22.694253  9996 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.696326 11607 ts_tablet_manager.cc:1398] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.019s	user 0.002s	sys 0.000s
I20240503 02:13:22.697757  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-5ba1f505010b403c91d3870cee19952a received from {username='slave'} at 127.0.145.7:43609
I20240503 02:13:22.697922  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-5ba1f505010b403c91d3870cee19952a on tablet 5ba1f505010b403c91d3870cee19952a with peer d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:22.703222  7907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.6:40397, user_credentials={real_user=slave}} blocked reactor thread for 122276us
I20240503 02:13:22.703739 11497 tablet_bootstrap.cc:492] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.704383 11497 tablet_bootstrap.cc:492] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap complete.
I20240503 02:13:22.705245 11687 tablet_replica.cc:330] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64: stopping tablet replica
I20240503 02:13:22.705463 11687 raft_consensus.cc:2229] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:22.705646 11687 raft_consensus.cc:2258] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.706853 11687 ts_tablet_manager.cc:1875] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:22.704576  9996 raft_consensus.cc:1203] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:22.705329  9995 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.714022 11687 ts_tablet_manager.cc:1888] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:22.714883 11687 log.cc:1196] T cab70b62bdc747a9b653bffde680f37e P 8d7c037d50c241b484ee843b06313a64: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/wals/cab70b62bdc747a9b653bffde680f37e
I20240503 02:13:22.715788  7817 catalog_manager.cc:4830] TS 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): tablet cab70b62bdc747a9b653bffde680f37e (table rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]) successfully deleted
I20240503 02:13:22.717001 11505 tablet_bootstrap.cc:492] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.717419 11505 tablet_bootstrap.cc:492] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap complete.
I20240503 02:13:22.717679 11505 ts_tablet_manager.cc:1367] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent bootstrapping tablet: real 0.382s	user 0.044s	sys 0.009s
I20240503 02:13:22.718436 11505 raft_consensus.cc:348] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:22.718724 11505 raft_consensus.cc:729] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Initialized, Role: LEARNER
I20240503 02:13:22.718931 11505 consensus_queue.cc:260] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:22.723088 11497 ts_tablet_manager.cc:1367] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent bootstrapping tablet: real 0.770s	user 0.045s	sys 0.011s
I20240503 02:13:22.724810 11497 raft_consensus.cc:348] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:22.725126 11497 raft_consensus.cc:729] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Initialized, Role: LEARNER
I20240503 02:13:22.725379 11497 consensus_queue.cc:260] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:22.726336 11497 ts_tablet_manager.cc:1398] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:13:22.727130  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-c23b06764ce54acb8388de94aea4cd4b received from {username='slave'} at 127.0.145.6:49885
I20240503 02:13:22.727298  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-c23b06764ce54acb8388de94aea4cd4b on tablet c23b06764ce54acb8388de94aea4cd4b with peer b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:22.728992 11505 ts_tablet_manager.cc:1398] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20240503 02:13:22.729732  7991 tablet_copy_service.cc:342] P d986fcfb926e47baa7d2d5b99f697334: Request end of tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-c45c5a14b85447c78fe2a27be4392c88 received from {username='slave'} at 127.0.145.8:52797
I20240503 02:13:22.729904  7991 tablet_copy_service.cc:433] P d986fcfb926e47baa7d2d5b99f697334: ending tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-c45c5a14b85447c78fe2a27be4392c88 on tablet c45c5a14b85447c78fe2a27be4392c88 with peer ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:22.736315  7950 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 with delete_type TABLET_DATA_TOMBSTONED (TS d986fcfb926e47baa7d2d5b99f697334 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34494
I20240503 02:13:22.737759 11597 tablet_replica.cc:330] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334: stopping tablet replica
I20240503 02:13:22.738005 11597 raft_consensus.cc:2229] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:22.738220 11597 pending_rounds.cc:62] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334: Trying to abort 1 pending ops.
I20240503 02:13:22.738278 11597 pending_rounds.cc:68] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023421040491200512 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } } }
I20240503 02:13:22.738656 11597 raft_consensus.cc:2894] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 02:13:22.738783 11597 raft_consensus.cc:2866] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:22.738899 11597 raft_consensus.cc:2258] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.740129 11597 ts_tablet_manager.cc:1875] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:22.751000 11083 tablet_copy_client.cc:308] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:22.751878  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 45fe6ec58eed40f1898d4a2f5727cfa1 from peer b85e85e631ab4e04a86bfe5a9f0aa512 ({username='slave'} at 127.0.145.6:56611)
I20240503 02:13:22.752081  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 45fe6ec58eed40f1898d4a2f5727cfa1 from peer b85e85e631ab4e04a86bfe5a9f0aa512 at {username='slave'} at 127.0.145.6:56611: session id = b85e85e631ab4e04a86bfe5a9f0aa512-45fe6ec58eed40f1898d4a2f5727cfa1
I20240503 02:13:22.755509  8122 tablet_copy_source_session.cc:205] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.755563 11597 ts_tablet_manager.cc:1888] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:22.755797 11597 log.cc:1196] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d986fcfb926e47baa7d2d5b99f697334: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/7e4acc8b5ceb4cd5b0f3708ff4f19298
I20240503 02:13:22.756780  7814 catalog_manager.cc:4830] TS d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 (table rebalance_test_table_6 [id=938b875abcf9426ead26522c081d6b7c]) successfully deleted
I20240503 02:13:22.757989 11083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45fe6ec58eed40f1898d4a2f5727cfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.766311  8231 consensus_queue.cc:237] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:22.769416 11083 tablet_copy_client.cc:783] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:22.769663  9995 raft_consensus.cc:1203] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:22.770721  8361 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:22.769682 11083 tablet_copy_client.cc:647] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:22.772822 11254 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:22.775091  9995 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:22.775813 11198 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:22.776669 11339 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.778355 11083 tablet_copy_client.cc:523] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:22.779922  8361 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.781183 10259 raft_consensus.cc:1203] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:22.782034  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 32e527fa9b2f441cb5668435681b4ff6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:22.783264 11083 tablet_bootstrap.cc:492] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap starting.
I20240503 02:13:22.783775  9996 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.793195  7824 catalog_manager.cc:5484] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:22.814396  8081 tablet_service.cc:1514] Processing DeleteTablet for tablet 32e527fa9b2f441cb5668435681b4ff6 with delete_type TABLET_DATA_TOMBSTONED (TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54670
I20240503 02:13:22.914213 11712 tablet_replica.cc:330] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: stopping tablet replica
I20240503 02:13:22.914778 11712 raft_consensus.cc:2229] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:22.915170 11712 raft_consensus.cc:2258] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.933177 10124 raft_consensus.cc:1203] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:22.952280  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:22.972697 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:22.973394  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:22.975643 11712 ts_tablet_manager.cc:1875] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:22.997673 11704 raft_consensus.cc:1050] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 to VOTER
I20240503 02:13:22.998642 11704 consensus_queue.cc:237] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:23.022413  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:23.022991  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:23.046774  9996 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.047668 10319 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.050374  8231 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.051424 10319 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.063045 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:23.065764  7967 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.066653 11728 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.097211 10319 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.104833  7821 catalog_manager.cc:5484] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:23.106540  8230 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.108561  9996 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.119627  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:23.135964 11684 raft_consensus.cc:1050] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334: attempting to promote NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 to VOTER
I20240503 02:13:23.136832 11684 consensus_queue.cc:237] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:23.137331 11712 ts_tablet_manager.cc:1888] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:23.137495 11712 log.cc:1196] T 32e527fa9b2f441cb5668435681b4ff6 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/32e527fa9b2f441cb5668435681b4ff6
I20240503 02:13:23.138392  7816 catalog_manager.cc:4830] TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): tablet 32e527fa9b2f441cb5668435681b4ff6 (table rebalance_test_table_1 [id=d4270f8900eb45459ee1e49c30ff6cff]) successfully deleted
I20240503 02:13:23.141027 10259 raft_consensus.cc:1203] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:23.144969  7971 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.146401  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:23.176332 11514 raft_consensus.cc:1050] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 to VOTER
I20240503 02:13:23.177475 11514 consensus_queue.cc:237] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:23.195459 11729 raft_consensus.cc:1050] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f to VOTER
I20240503 02:13:23.196317 11729 consensus_queue.cc:237] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:23.206163 11083 tablet_bootstrap.cc:492] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:23.206712 11083 tablet_bootstrap.cc:492] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap complete.
I20240503 02:13:23.208230  9996 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.208932 11516 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.209235 11083 ts_tablet_manager.cc:1367] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent bootstrapping tablet: real 0.426s	user 0.075s	sys 0.043s
I20240503 02:13:23.210143 11083 raft_consensus.cc:348] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:23.210420 11083 raft_consensus.cc:729] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Initialized, Role: LEARNER
I20240503 02:13:23.210685 11083 consensus_queue.cc:260] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:23.213356  7967 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.213447 11083 ts_tablet_manager.cc:1398] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:13:23.214012 11516 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.215152  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-45fe6ec58eed40f1898d4a2f5727cfa1 received from {username='slave'} at 127.0.145.6:56611
I20240503 02:13:23.215278  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-45fe6ec58eed40f1898d4a2f5727cfa1 on tablet 45fe6ec58eed40f1898d4a2f5727cfa1 with peer b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:23.217970  8230 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.218935 11518 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.239888 11516 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.250674  9996 raft_consensus.cc:1203] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:23.252976  8231 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.255869  9864 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:23.256181  7968 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:23.256834 11704 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:23.257825 11704 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:23.258209 10124 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:23.258935 11704 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:23.268077  8361 raft_consensus.cc:1261] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.268841 11400 consensus_queue.cc:1035] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.275941  9996 raft_consensus.cc:1261] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.275923  7971 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.276556 11400 consensus_queue.cc:1035] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.291786  8230 raft_consensus.cc:1261] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.295074 11684 consensus_queue.cc:1035] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.295670  9995 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.299371 11704 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:23.304525  7971 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:23.308602  9864 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:23.309511  7829 catalog_manager.cc:5484] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:23.320302  7822 catalog_manager.cc:5484] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 7 to 8, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:23.335680 11400 raft_consensus.cc:2932] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.336766 10124 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:23.341246  8361 raft_consensus.cc:2932] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.342679  9994 raft_consensus.cc:2932] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.347584  7822 catalog_manager.cc:5484] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:23.374565 11518 raft_consensus.cc:1050] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f to VOTER
I20240503 02:13:23.375751 11518 consensus_queue.cc:237] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:23.378901 11339 raft_consensus.cc:1050] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 to VOTER
I20240503 02:13:23.380164 11339 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:23.381596 10125 raft_consensus.cc:1261] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.382874  7968 raft_consensus.cc:1261] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.383227  8231 raft_consensus.cc:1261] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.383183 11516 consensus_queue.cc:1035] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.387693 10259 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.388005 11516 consensus_queue.cc:1035] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.389389 11516 consensus_queue.cc:1035] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.390552  8361 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.399796  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:23.419857 11770 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.421809  8100 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.430754 11339 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.440384 11772 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.464771 11518 raft_consensus.cc:2932] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:23.466140 10125 raft_consensus.cc:2932] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:23.473843  8232 raft_consensus.cc:2932] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:23.478912  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:23.480094  7971 consensus_queue.cc:237] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:23.481103  7823 catalog_manager.cc:5484] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:23.483842  8231 raft_consensus.cc:2932] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.484727  9995 raft_consensus.cc:1261] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:23.485553 11683 consensus_queue.cc:1035] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:23.490747  7971 raft_consensus.cc:2932] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:23.491526 11683 raft_consensus.cc:2932] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.495673  9995 raft_consensus.cc:2932] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.498965  8232 raft_consensus.cc:1261] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:23.500425 11400 consensus_queue.cc:1035] T f231fe5dfd9e4c87b90bad75818a7060 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:23.501988 11772 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:23.505126  7822 catalog_manager.cc:5484] T f231fe5dfd9e4c87b90bad75818a7060 P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 5 to 6, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.506690  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f231fe5dfd9e4c87b90bad75818a7060 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:23.509049  8341 tablet_service.cc:1514] Processing DeleteTablet for tablet f231fe5dfd9e4c87b90bad75818a7060 with delete_type TABLET_DATA_TOMBSTONED (TS 8d7c037d50c241b484ee843b06313a64 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37080
I20240503 02:13:23.508915 10259 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:23.511129  8361 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:23.516340  7823 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:23.530695 11786 tablet_replica.cc:330] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64: stopping tablet replica
I20240503 02:13:23.531672 11786 raft_consensus.cc:2229] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:23.532552 11786 raft_consensus.cc:2258] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:23.535122 11786 ts_tablet_manager.cc:1875] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:23.534348  8231 raft_consensus.cc:2932] T f231fe5dfd9e4c87b90bad75818a7060 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.560688 11786 ts_tablet_manager.cc:1888] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:23.563035 11786 log.cc:1196] T f231fe5dfd9e4c87b90bad75818a7060 P 8d7c037d50c241b484ee843b06313a64: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/wals/f231fe5dfd9e4c87b90bad75818a7060
I20240503 02:13:23.564186  7817 catalog_manager.cc:4830] TS 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): tablet f231fe5dfd9e4c87b90bad75818a7060 (table rebalance_test_table_6 [id=938b875abcf9426ead26522c081d6b7c]) successfully deleted
I20240503 02:13:23.575816  8099 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:23.593518  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:23.635129  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:23.670877  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:23.678072  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:23.710368  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:23.712495  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:23.733273 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:23.741452 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:23.755546 11704 raft_consensus.cc:1050] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 to VOTER
I20240503 02:13:23.756474 11704 consensus_queue.cc:237] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:23.764233  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:23.765229  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:23.782138 11684 raft_consensus.cc:1050] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334: attempting to promote NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 to VOTER
I20240503 02:13:23.783576 11684 consensus_queue.cc:237] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:23.795379 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:23.800415 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:23.840760  9994 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.841759 11729 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.848129  8231 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.850315 10259 raft_consensus.cc:1261] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.851996 11136 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.852941  8100 raft_consensus.cc:1261] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.859019  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:23.859684  8361 raft_consensus.cc:1261] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.860396 11400 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.866307 11400 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.868892  7971 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:23.869637 11704 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.876060 11400 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:23.886780  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:23.889446 11729 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.895735  9994 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.897303  7822 catalog_manager.cc:5484] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:23.895782  8231 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.912305  7968 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.920238  8100 consensus_queue.cc:237] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:23.924419  7968 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:23.926985 11704 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:23.933005 11400 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:23.940421 11704 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.950879  8361 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:23.951228  8100 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:23.952561  7821 catalog_manager.cc:5484] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:23.962078  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 45fe6ec58eed40f1898d4a2f5727cfa1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:23.963934  7971 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:23.980185  7821 catalog_manager.cc:5484] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 5 to 6, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:23.984735  8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 45fe6ec58eed40f1898d4a2f5727cfa1 with delete_type TABLET_DATA_TOMBSTONED (TS 2942efc0ecb448d2ba2f6bca20fbb29c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35426
I20240503 02:13:24.006989  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:24.009912 10259 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:24.024006 11833 tablet_replica.cc:330] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: stopping tablet replica
I20240503 02:13:24.024399 11833 raft_consensus.cc:2229] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:24.024755 11833 raft_consensus.cc:2258] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.028141 11833 ts_tablet_manager.cc:1875] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:24.044378  9995 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:24.046255 11704 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20240503 02:13:24.062194 11833 ts_tablet_manager.cc:1888] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:24.065207  9994 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:24.091321  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:24.091799 11833 log.cc:1196] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/45fe6ec58eed40f1898d4a2f5727cfa1
I20240503 02:13:24.092734  7814 catalog_manager.cc:4830] TS 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): tablet 45fe6ec58eed40f1898d4a2f5727cfa1 (table rebalance_test_table_9 [id=5f6139f12f9b4ab18ce35a8831b39150]) successfully deleted
I20240503 02:13:24.101601  7971 consensus_queue.cc:237] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:24.104673 10259 raft_consensus.cc:1261] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:24.105340  8100 raft_consensus.cc:1261] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:24.105275 11684 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:24.106283 11684 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:24.108860 11684 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:24.113252  8100 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:24.121084  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c45c5a14b85447c78fe2a27be4392c88 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:24.122851 10259 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:24.123273  7822 catalog_manager.cc:5484] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 5 to 6, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:24.129490  8341 tablet_service.cc:1514] Processing DeleteTablet for tablet c45c5a14b85447c78fe2a27be4392c88 with delete_type TABLET_DATA_TOMBSTONED (TS 8d7c037d50c241b484ee843b06313a64 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37080
I20240503 02:13:24.145579  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:24.188257 11841 tablet_replica.cc:330] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: stopping tablet replica
I20240503 02:13:24.188534 11841 raft_consensus.cc:2229] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:24.188795 11841 raft_consensus.cc:2258] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.196676  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:24.203289 11841 ts_tablet_manager.cc:1875] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:24.203460  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:24.237048 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:24.284920 11841 ts_tablet_manager.cc:1888] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:24.287209 11841 log.cc:1196] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/wals/c45c5a14b85447c78fe2a27be4392c88
I20240503 02:13:24.290006 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:24.291344  7817 catalog_manager.cc:4830] TS 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): tablet c45c5a14b85447c78fe2a27be4392c88 (table rebalance_test_table_6 [id=938b875abcf9426ead26522c081d6b7c]) successfully deleted
I20240503 02:13:24.318233  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:24.352523  8361 consensus_queue.cc:237] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:24.362700 10259 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:24.363063  8231 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:24.363785 11516 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:24.366863  8099 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:24.370143 11518 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:24.373279 11516 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:24.434222 11518 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:24.443537 10260 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:24.443639  8099 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:24.440192  8231 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:24.449394  7821 catalog_manager.cc:5484] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:24.458019  8361 consensus_queue.cc:237] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
W20240503 02:13:24.465658  8295 consensus_peers.cc:485] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de9c2a3396864696af88be1ca12ee0f1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:24.467867 10260 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:24.468871 11857 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:24.469918  8100 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:24.473901 11516 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:24.485606 11857 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:24.490129  8100 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:24.495893  7817 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet de9c2a3396864696af88be1ca12ee0f1 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:13:24.498395  7822 catalog_manager.cc:5484] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 6 to 7, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:24.504354  8231 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:24.507375 11516 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:24.509151  8230 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:24.554075 10260 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:24.574388  8360 consensus_queue.cc:237] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:24.579057 10260 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:24.579155  8100 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:24.580168 11518 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:24.580857 11518 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:13:24.585100  8295 consensus_peers.cc:485] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de9c2a3396864696af88be1ca12ee0f1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:24.594810 11516 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:24.600656 10260 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:24.609673  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet de9c2a3396864696af88be1ca12ee0f1 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:24.614195  8100 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:24.621107  7822 catalog_manager.cc:5484] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:24.626346  8211 tablet_service.cc:1514] Processing DeleteTablet for tablet de9c2a3396864696af88be1ca12ee0f1 with delete_type TABLET_DATA_TOMBSTONED (TS 2942efc0ecb448d2ba2f6bca20fbb29c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35426
I20240503 02:13:24.637394  8231 consensus_queue.cc:237] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:24.654734 11877 tablet_replica.cc:330] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c: stopping tablet replica
I20240503 02:13:24.655793  8100 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:24.655709  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2448e12efbd44a90b1f8bce9519e73e0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:24.656476 11770 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:24.658077 11877 raft_consensus.cc:2229] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:24.658787 11877 raft_consensus.cc:2258] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.659890  8361 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:24.660473 11770 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:24.661774 11877 ts_tablet_manager.cc:1875] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:24.663636 11770 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:24.680980 11877 ts_tablet_manager.cc:1888] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 02:13:24.681221 11877 log.cc:1196] T de9c2a3396864696af88be1ca12ee0f1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/de9c2a3396864696af88be1ca12ee0f1
I20240503 02:13:24.702913  8361 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:24.706926  8100 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:24.708774  7814 catalog_manager.cc:4830] TS 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): tablet de9c2a3396864696af88be1ca12ee0f1 (table rebalance_test_table_7 [id=635c8841bff54466a804d93ade171b2f]) successfully deleted
I20240503 02:13:24.710371  7822 catalog_manager.cc:5484] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index -1 to 4, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:24.778074  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:24.823434  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:24.838444  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:24.855893  8361 consensus_queue.cc:237] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:24.859936  7971 raft_consensus.cc:1261] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:24.860041  8231 raft_consensus.cc:1261] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:24.860852 11857 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:24.861516 11857 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:24.871850  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:13:24.882249  8297 consensus_peers.cc:485] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 834aaba965c8443c862e98a070e36cac. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:24.888473 11516 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:24.889304  7968 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:24.900007  7823 catalog_manager.cc:5484] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:24.938165 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:24.970325  8231 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:24.992527  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:25.007699 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:25.008301  8231 consensus_queue.cc:237] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:25.062842  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:13:25.091373  8165 consensus_peers.cc:485] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecb5c78a22a74728ab40f1da8f90388c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:25.092239  7971 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:25.092941 11888 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:25.098335  8100 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:25.099066 11888 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:25.108965 11888 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:25.110577  8099 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:25.112663 11952 ts_tablet_manager.cc:902] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:25.137629  7823 catalog_manager.cc:5484] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:25.165830  7971 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:25.199211 11952 tablet_copy_client.cc:308] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:25.200357  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet de9c2a3396864696af88be1ca12ee0f1 from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:43609)
I20240503 02:13:25.200611  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet de9c2a3396864696af88be1ca12ee0f1 from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:43609: session id = d826d4dae2aa4e05b2f64c9b17a1126f-de9c2a3396864696af88be1ca12ee0f1
I20240503 02:13:25.207175 11972 ts_tablet_manager.cc:902] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:25.208966  8382 tablet_copy_source_session.cc:205] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:25.217429 11952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de9c2a3396864696af88be1ca12ee0f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:25.247584  8100 consensus_queue.cc:237] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:25.257587  8361 consensus_queue.cc:237] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:25.258517  8231 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:25.259328 11729 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:25.259892 10260 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:25.260200 11952 tablet_copy_client.cc:783] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:25.260507 11952 tablet_copy_client.cc:647] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:25.261404 10124 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:25.262292  8099 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:25.262847 11516 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:25.263203 11866 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:25.267277 11729 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:25.287144  8361 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:25.288002 11729 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:25.310712 11729 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:25.311676 10260 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:25.316762  8361 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:25.321053  7822 catalog_manager.cc:5484] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:25.326320 11516 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:25.331565 10125 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:25.334879 11972 tablet_copy_client.cc:308] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:25.335675  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet 2448e12efbd44a90b1f8bce9519e73e0 from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:37535)
I20240503 02:13:25.335876  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet 2448e12efbd44a90b1f8bce9519e73e0 from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:37535: session id = 03f99420df704238ab300ad6e58d878c-2448e12efbd44a90b1f8bce9519e73e0
I20240503 02:13:25.352335  8252 tablet_copy_source_session.cc:205] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:25.353273  8099 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:25.363580 11972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2448e12efbd44a90b1f8bce9519e73e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:25.363934  7822 catalog_manager.cc:5484] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:25.368189  8298 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e4acc8b5ceb4cd5b0f3708ff4f19298. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:25.368989 11952 tablet_copy_client.cc:523] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:25.368750  8100 consensus_queue.cc:237] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:25.373131 11952 tablet_bootstrap.cc:492] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
W20240503 02:13:25.374627  7816 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet b64f66ace86e42f3ac213ba85bcab584 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
W20240503 02:13:25.382942 11989 log.cc:924] Time spent T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20240503 02:13:25.382575  8231 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:25.386986 12006 ts_tablet_manager.cc:902] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
W20240503 02:13:25.392189  7904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.7:39415, user_credentials={real_user=slave}} blocked reactor thread for 49082.9us
I20240503 02:13:25.401816 11972 tablet_copy_client.cc:783] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
W20240503 02:13:25.404999  8038 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b64f66ace86e42f3ac213ba85bcab584. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:25.405639 10260 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:25.406536 11729 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:25.412561  8361 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:25.414149 11729 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:25.416290 11972 tablet_copy_client.cc:647] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:25.419374 11972 tablet_copy_client.cc:523] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:25.420818 11729 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:25.423365 10260 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:25.452476 12006 tablet_copy_client.cc:308] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:25.453750  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet 834aaba965c8443c862e98a070e36cac from peer ca05c9a8bee148dab8a2a982b8fa2ea4 ({username='slave'} at 127.0.145.8:38819)
I20240503 02:13:25.453995  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet 834aaba965c8443c862e98a070e36cac from peer ca05c9a8bee148dab8a2a982b8fa2ea4 at {username='slave'} at 127.0.145.8:38819: session id = ca05c9a8bee148dab8a2a982b8fa2ea4-834aaba965c8443c862e98a070e36cac
I20240503 02:13:25.458410  8382 tablet_copy_source_session.cc:205] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:25.459375  7822 catalog_manager.cc:5484] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:25.468849  7816 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b64f66ace86e42f3ac213ba85bcab584 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:13:25.471428 12006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 834aaba965c8443c862e98a070e36cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:25.471776  8361 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:25.479436 11972 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
W20240503 02:13:25.481092  7816 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet b64f66ace86e42f3ac213ba85bcab584 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 6 but the committed config has opid_index of 7
I20240503 02:13:25.522440 12027 ts_tablet_manager.cc:902] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:25.562918  8231 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:25.564971 11957 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:25.570981  8231 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:25.589354 12006 tablet_copy_client.cc:783] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:25.589679 12006 tablet_copy_client.cc:647] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:25.598289 12006 tablet_copy_client.cc:523] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:25.600008  8361 consensus_queue.cc:237] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:25.638041  8231 consensus_queue.cc:237] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
W20240503 02:13:25.638330  8295 consensus_peers.cc:485] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 402de43677c44deab8793da1895d4fb2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:25.639174  8230 raft_consensus.cc:1261] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:25.639895 11518 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:25.641779  8167 consensus_peers.cc:485] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f14607425da84bb4b162a5789daa844c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:25.642555 10125 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:25.643456  8100 raft_consensus.cc:1261] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:25.644026 11888 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:25.645174 11518 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:25.646209  8361 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:25.647991 11888 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240503 02:13:25.671269 11888 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:25.674270 12006 tablet_bootstrap.cc:492] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap starting.
I20240503 02:13:25.675442 11518 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:25.683195  8361 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:25.685274 10125 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:25.698791  8100 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:25.716552  7823 catalog_manager.cc:5484] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:25.727965  7822 catalog_manager.cc:5484] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: config changed from index 8 to 9, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:25.731475  8230 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:25.748790 12067 ts_tablet_manager.cc:902] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:25.795485 12027 tablet_copy_client.cc:308] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:25.796397  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet ecb5c78a22a74728ab40f1da8f90388c from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:54377)
I20240503 02:13:25.796628  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet ecb5c78a22a74728ab40f1da8f90388c from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:54377: session id = d826d4dae2aa4e05b2f64c9b17a1126f-ecb5c78a22a74728ab40f1da8f90388c
I20240503 02:13:25.800540  8252 tablet_copy_source_session.cc:205] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:25.803676 12027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb5c78a22a74728ab40f1da8f90388c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:25.811553  8231 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:25.813588 12067 tablet_copy_client.cc:308] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:25.818632 12027 tablet_copy_client.cc:783] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:25.818933 12027 tablet_copy_client.cc:647] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:25.825206 11972 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:25.825817 11972 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:25.826190 11972 ts_tablet_manager.cc:1367] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.347s	user 0.050s	sys 0.014s
I20240503 02:13:25.827287 11972 raft_consensus.cc:348] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:25.827642 11972 raft_consensus.cc:729] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:25.827909 11972 consensus_queue.cc:260] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:25.830513 12027 tablet_copy_client.cc:523] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:25.835673 12027 tablet_bootstrap.cc:492] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
I20240503 02:13:25.847014  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:25.854074  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:25.857375  8360 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:25.857604  8099 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:25.858222 11888 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:25.858872 11889 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:25.861186 10260 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:25.861744 11889 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:25.862584  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:25.905496  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:25.907105  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:25.908963  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:42737)
I20240503 02:13:25.909195  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:42737: session id = 03f99420df704238ab300ad6e58d878c-7e4acc8b5ceb4cd5b0f3708ff4f19298
W20240503 02:13:25.912058 11952 log.cc:924] Time spent T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f: Append to log took a long time: real 0.137s	user 0.004s	sys 0.000s
I20240503 02:13:25.912420  8382 tablet_copy_source_session.cc:205] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:25.914637 11952 tablet_bootstrap.cc:492] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:25.914654 12067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:25.923399  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:25.926688 12067 tablet_copy_client.cc:783] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:25.927121 12067 tablet_copy_client.cc:647] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:25.932137 11952 tablet_bootstrap.cc:492] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:13:25.938925 11952 ts_tablet_manager.cc:1367] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 0.566s	user 0.059s	sys 0.012s
I20240503 02:13:25.943377 11952 raft_consensus.cc:348] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:25.945343 11952 raft_consensus.cc:729] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:13:25.947034 11952 consensus_queue.cc:260] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:25.956962  8099 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:25.961732 12114 ts_tablet_manager.cc:902] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:25.962399 12067 tablet_copy_client.cc:523] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:25.964072 11952 ts_tablet_manager.cc:1398] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.025s	user 0.000s	sys 0.002s
I20240503 02:13:25.970769 12067 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
I20240503 02:13:25.977945  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-de9c2a3396864696af88be1ca12ee0f1 received from {username='slave'} at 127.0.145.7:43609
I20240503 02:13:25.978122  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-de9c2a3396864696af88be1ca12ee0f1 on tablet de9c2a3396864696af88be1ca12ee0f1 with peer d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:25.988997  8038 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f44b2b53de84a87a8757c0d00bfed37. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:25.989059 10259 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:13:25.989671 11957 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:13:25.996524  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:25.997581 11889 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:26.009011  8360 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:26.010411 10261 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:26.011266  8100 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:26.024415  7821 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:26.040954  8231 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:26.060948  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:26.062363 10260 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:26.063133 11889 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:26.071094  9864 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:13:26.072950 11889 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:26.073447  8360 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:26.074760 11888 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:26.076699 10261 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:26.080878  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e966caa651e4e42890be5d79f851a1f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:26.083051  7821 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 6 to 7, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:26.085054 11957 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:13:26.096153  8081 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e966caa651e4e42890be5d79f851a1f with delete_type TABLET_DATA_TOMBSTONED (TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54670
I20240503 02:13:26.096309  8231 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } }
W20240503 02:13:26.100822  8165 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Couldn't send request to peer d986fcfb926e47baa7d2d5b99f697334. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e966caa651e4e42890be5d79f851a1f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:26.092114  8360 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:26.106577 11957 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:26.126318  7822 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 11 to 12, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:26.133379 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:26.141664 10259 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:26.144110  8361 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:26.145036 11889 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:26.147779 11892 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } } }
I20240503 02:13:26.149889  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:26.152642  7814 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7e966caa651e4e42890be5d79f851a1f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:13:26.156231  8361 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } } }
I20240503 02:13:26.158905 11888 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:26.162366  7828 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 7 to 8, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:26.174100 10259 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:26.180802 11952 ts_tablet_manager.cc:902] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:26.195101 10261 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } } }
I20240503 02:13:26.198820 11972 ts_tablet_manager.cc:1398] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.372s	user 0.015s	sys 0.027s
I20240503 02:13:26.199787  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-2448e12efbd44a90b1f8bce9519e73e0 received from {username='slave'} at 127.0.145.5:37535
I20240503 02:13:26.199940  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session 03f99420df704238ab300ad6e58d878c-2448e12efbd44a90b1f8bce9519e73e0 on tablet 2448e12efbd44a90b1f8bce9519e73e0 with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:26.203845 12135 ts_tablet_manager.cc:902] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:26.223609  8099 consensus_queue.cc:237] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
W20240503 02:13:26.227377  8038 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b64f66ace86e42f3ac213ba85bcab584. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:26.227377  8360 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:26.228070 11957 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:26.230890 12017 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:26.232646 10260 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:26.234434  8361 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:26.236902 12114 tablet_copy_client.cc:308] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:26.237617  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet b64f66ace86e42f3ac213ba85bcab584 from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:48749)
I20240503 02:13:26.237788  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet b64f66ace86e42f3ac213ba85bcab584 from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:48749: session id = 03f99420df704238ab300ad6e58d878c-b64f66ace86e42f3ac213ba85bcab584
I20240503 02:13:26.240219 11957 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:26.240891  7823 catalog_manager.cc:5484] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:26.245990  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b64f66ace86e42f3ac213ba85bcab584 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:26.246745 12130 tablet_replica.cc:330] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: stopping tablet replica
I20240503 02:13:26.247094 12130 raft_consensus.cc:2229] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:26.247503 12130 raft_consensus.cc:2258] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:26.253161 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:26.251330 10260 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:26.256898  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:26.278287  8360 consensus_queue.cc:237] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
W20240503 02:13:26.281973  8295 consensus_peers.cc:485] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 402de43677c44deab8793da1895d4fb2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:26.282894  8231 raft_consensus.cc:1261] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:26.283072  8100 raft_consensus.cc:1261] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:13:26.284025  8298 consensus_peers.cc:485] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 402de43677c44deab8793da1895d4fb2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:26.284251 11866 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:26.285521 11866 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:26.287101 11866 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:26.291790  8211 tablet_service.cc:1514] Processing DeleteTablet for tablet b64f66ace86e42f3ac213ba85bcab584 with delete_type TABLET_DATA_TOMBSTONED (TS 2942efc0ecb448d2ba2f6bca20fbb29c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35426
I20240503 02:13:26.292304  8231 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:26.296031  8099 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:26.297806 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:26.301244  8122 tablet_copy_source_session.cc:205] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.303810 12114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64f66ace86e42f3ac213ba85bcab584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.306277  7823 catalog_manager.cc:5484] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 4 to 5, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:26.312490 12114 tablet_copy_client.cc:783] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.313351 12114 tablet_copy_client.cc:647] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.315536  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:26.317618 12114 tablet_copy_client.cc:523] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.325165 12114 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
I20240503 02:13:26.325667 12155 tablet_replica.cc:330] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c: stopping tablet replica
I20240503 02:13:26.325762  9864 raft_consensus.cc:1203] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:26.325981 12155 raft_consensus.cc:2229] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:26.326231 12155 raft_consensus.cc:2258] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:26.334650 12130 ts_tablet_manager.cc:1875] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:26.337785  9863 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:26.342725 11952 tablet_copy_client.cc:308] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:26.343552  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet 402de43677c44deab8793da1895d4fb2 from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:43609)
I20240503 02:13:26.343732  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet 402de43677c44deab8793da1895d4fb2 from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:43609: session id = d826d4dae2aa4e05b2f64c9b17a1126f-402de43677c44deab8793da1895d4fb2
I20240503 02:13:26.346930  8382 tablet_copy_source_session.cc:205] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.351491 11952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 402de43677c44deab8793da1895d4fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.389926 12130 ts_tablet_manager.cc:1888] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:13:26.390676 10125 raft_consensus.cc:1203] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 02:13:26.394273 12155 ts_tablet_manager.cc:1875] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:26.395789  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:26.399564 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:26.408892 11952 tablet_copy_client.cc:783] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.409212 11952 tablet_copy_client.cc:647] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.411006 11952 tablet_copy_client.cc:523] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.411963 12027 tablet_bootstrap.cc:492] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:26.412492 12027 tablet_bootstrap.cc:492] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:13:26.438484 12027 ts_tablet_manager.cc:1367] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 0.577s	user 0.045s	sys 0.016s
I20240503 02:13:26.453707 11952 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
I20240503 02:13:26.461196 12027 raft_consensus.cc:348] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:26.461544 12027 raft_consensus.cc:729] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:13:26.461736 12130 log.cc:1196] T 7e966caa651e4e42890be5d79f851a1f P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/7e966caa651e4e42890be5d79f851a1f
I20240503 02:13:26.461778 12027 consensus_queue.cc:260] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:26.462396 12027 ts_tablet_manager.cc:1398] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.024s	user 0.001s	sys 0.000s
I20240503 02:13:26.464128  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-ecb5c78a22a74728ab40f1da8f90388c received from {username='slave'} at 127.0.145.7:54377
I20240503 02:13:26.464269  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-ecb5c78a22a74728ab40f1da8f90388c on tablet ecb5c78a22a74728ab40f1da8f90388c with peer d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:26.470942  7816 catalog_manager.cc:4830] TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): tablet 7e966caa651e4e42890be5d79f851a1f (table rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]) successfully deleted
I20240503 02:13:26.501017 12067 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:26.502197 12067 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:26.503285 12067 ts_tablet_manager.cc:1367] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.533s	user 0.057s	sys 0.012s
I20240503 02:13:26.505386 12067 raft_consensus.cc:348] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:26.506531 12067 raft_consensus.cc:729] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:26.506928 12067 consensus_queue.cc:260] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:26.509218 12067 ts_tablet_manager.cc:1398] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:13:26.510488  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-7e4acc8b5ceb4cd5b0f3708ff4f19298 received from {username='slave'} at 127.0.145.5:42737
I20240503 02:13:26.510648  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session 03f99420df704238ab300ad6e58d878c-7e4acc8b5ceb4cd5b0f3708ff4f19298 on tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:26.514803  8099 consensus_queue.cc:237] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
W20240503 02:13:26.522404  8037 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 007b59de142c4b95966494ba097b1a09. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:26.523949  8231 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:26.524013  7971 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:26.524865 12151 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:26.525569 12152 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:26.531643 12152 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:26.540158  7823 catalog_manager.cc:5484] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:26.554760 12135 tablet_copy_client.cc:308] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:26.566144  8231 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:26.573545 12181 ts_tablet_manager.cc:902] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:26.577482  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet f14607425da84bb4b162a5789daa844c from peer ca05c9a8bee148dab8a2a982b8fa2ea4 ({username='slave'} at 127.0.145.8:44853)
I20240503 02:13:26.577682  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet f14607425da84bb4b162a5789daa844c from peer ca05c9a8bee148dab8a2a982b8fa2ea4 at {username='slave'} at 127.0.145.8:44853: session id = ca05c9a8bee148dab8a2a982b8fa2ea4-f14607425da84bb4b162a5789daa844c
I20240503 02:13:26.591102  8252 tablet_copy_source_session.cc:205] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.591753 12155 ts_tablet_manager.cc:1888] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 02:13:26.591957 12155 log.cc:1196] T b64f66ace86e42f3ac213ba85bcab584 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/b64f66ace86e42f3ac213ba85bcab584
I20240503 02:13:26.592648 12135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14607425da84bb4b162a5789daa844c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.601215  7814 catalog_manager.cc:4830] TS 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): tablet b64f66ace86e42f3ac213ba85bcab584 (table rebalance_test_table_9 [id=5f6139f12f9b4ab18ce35a8831b39150]) successfully deleted
I20240503 02:13:26.601539 12135 tablet_copy_client.cc:783] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.601826 12135 tablet_copy_client.cc:647] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.603935 12135 tablet_copy_client.cc:523] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.616462 12135 tablet_bootstrap.cc:492] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap starting.
I20240503 02:13:26.631549  7971 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:26.635238  9863 raft_consensus.cc:1203] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:13:26.668195  8231 consensus_queue.cc:237] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:26.682377 12114 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:26.682888 12114 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:26.683205 12114 ts_tablet_manager.cc:1367] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.358s	user 0.060s	sys 0.001s
I20240503 02:13:26.684082 12114 raft_consensus.cc:348] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:26.684370 12114 raft_consensus.cc:729] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:26.684576 12114 consensus_queue.cc:260] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:26.688838 12114 ts_tablet_manager.cc:1398] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240503 02:13:26.689646  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-b64f66ace86e42f3ac213ba85bcab584 received from {username='slave'} at 127.0.145.5:48749
I20240503 02:13:26.689808  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session 03f99420df704238ab300ad6e58d878c-b64f66ace86e42f3ac213ba85bcab584 on tablet b64f66ace86e42f3ac213ba85bcab584 with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:26.693107  8360 raft_consensus.cc:1261] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:26.693869 11888 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:26.696986 12114 tablet_replica.cc:330] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: stopping tablet replica
I20240503 02:13:26.697719 12114 raft_consensus.cc:2229] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Raft consensus shutting down.
I20240503 02:13:26.698032 12114 raft_consensus.cc:2258] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Raft consensus is shut down!
I20240503 02:13:26.699231 12192 ts_tablet_manager.cc:902] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:26.701830 12114 ts_tablet_manager.cc:1875] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:26.703711  7971 raft_consensus.cc:1261] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:26.703648  8165 consensus_peers.cc:485] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6043da7d2e584af885ff6a5bf8f06912. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:26.704555 11888 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:26.711022 12006 tablet_bootstrap.cc:492] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:26.712638 11888 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:26.714972  8361 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:26.718483 12114 ts_tablet_manager.cc:1888] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:26.718736 12114 log.cc:1196] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/wals/b64f66ace86e42f3ac213ba85bcab584
I20240503 02:13:26.719282 12114 ts_tablet_manager.cc:902] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:26.723912 12192 tablet_copy_client.cc:308] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:26.724450  7823 catalog_manager.cc:5484] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:26.725072  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet 7e966caa651e4e42890be5d79f851a1f from peer d986fcfb926e47baa7d2d5b99f697334 ({username='slave'} at 127.0.145.1:39931)
I20240503 02:13:26.725252  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet 7e966caa651e4e42890be5d79f851a1f from peer d986fcfb926e47baa7d2d5b99f697334 at {username='slave'} at 127.0.145.1:39931: session id = d986fcfb926e47baa7d2d5b99f697334-7e966caa651e4e42890be5d79f851a1f
I20240503 02:13:26.725986 11952 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:26.726475 11952 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:13:26.726840 11952 ts_tablet_manager.cc:1367] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 0.273s	user 0.050s	sys 0.007s
I20240503 02:13:26.727747 11952 raft_consensus.cc:348] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:26.728058  8252 tablet_copy_source_session.cc:205] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.728065 11952 raft_consensus.cc:729] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:13:26.728312 11952 consensus_queue.cc:260] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:26.735816 11952 ts_tablet_manager.cc:1398] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.000s
I20240503 02:13:26.736656  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-402de43677c44deab8793da1895d4fb2 received from {username='slave'} at 127.0.145.7:43609
I20240503 02:13:26.736837  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-402de43677c44deab8793da1895d4fb2 on tablet 402de43677c44deab8793da1895d4fb2 with peer d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:26.740080 12192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e966caa651e4e42890be5d79f851a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.744830 10125 raft_consensus.cc:1203] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:26.752089 12006 tablet_bootstrap.cc:492] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap complete.
I20240503 02:13:26.760560 12114 tablet_copy_client.cc:308] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:26.762504  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet b64f66ace86e42f3ac213ba85bcab584 from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:48749)
I20240503 02:13:26.762792  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet b64f66ace86e42f3ac213ba85bcab584 from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:48749: session id = 03f99420df704238ab300ad6e58d878c-b64f66ace86e42f3ac213ba85bcab584
I20240503 02:13:26.767436 12067 ts_tablet_manager.cc:902] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:26.768550  8122 tablet_copy_source_session.cc:205] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.771221 12114 ts_tablet_manager.cc:1875] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:13:26.779790 12181 tablet_copy_client.cc:308] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:26.780997  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 0f44b2b53de84a87a8757c0d00bfed37 from peer b85e85e631ab4e04a86bfe5a9f0aa512 ({username='slave'} at 127.0.145.6:56611)
I20240503 02:13:26.781194  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 0f44b2b53de84a87a8757c0d00bfed37 from peer b85e85e631ab4e04a86bfe5a9f0aa512 at {username='slave'} at 127.0.145.6:56611: session id = b85e85e631ab4e04a86bfe5a9f0aa512-0f44b2b53de84a87a8757c0d00bfed37
I20240503 02:13:26.782735 12006 ts_tablet_manager.cc:1367] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent bootstrapping tablet: real 1.109s	user 0.050s	sys 0.009s
I20240503 02:13:26.783702 12006 raft_consensus.cc:348] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:26.783991 12006 raft_consensus.cc:729] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Initialized, Role: LEARNER
I20240503 02:13:26.784211 12006 consensus_queue.cc:260] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:26.785082  8122 tablet_copy_source_session.cc:205] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.787001 12181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f44b2b53de84a87a8757c0d00bfed37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.800506 12114 ts_tablet_manager.cc:1888] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240503 02:13:26.800905 12114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64f66ace86e42f3ac213ba85bcab584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.801678 12135 tablet_bootstrap.cc:492] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:26.802189 12135 tablet_bootstrap.cc:492] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap complete.
I20240503 02:13:26.802407 12006 ts_tablet_manager.cc:1398] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent starting tablet: real 0.019s	user 0.002s	sys 0.000s
I20240503 02:13:26.802546 12135 ts_tablet_manager.cc:1367] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent bootstrapping tablet: real 0.190s	user 0.058s	sys 0.009s
I20240503 02:13:26.803524 12135 raft_consensus.cc:348] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:26.803808 12135 raft_consensus.cc:729] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Initialized, Role: LEARNER
I20240503 02:13:26.804010 12135 consensus_queue.cc:260] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:26.804582 12135 ts_tablet_manager.cc:1398] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:13:26.805316  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-f14607425da84bb4b162a5789daa844c received from {username='slave'} at 127.0.145.8:44853
I20240503 02:13:26.805454  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-f14607425da84bb4b162a5789daa844c on tablet f14607425da84bb4b162a5789daa844c with peer ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:26.805604  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-834aaba965c8443c862e98a070e36cac received from {username='slave'} at 127.0.145.8:38819
I20240503 02:13:26.805765  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-834aaba965c8443c862e98a070e36cac on tablet 834aaba965c8443c862e98a070e36cac with peer ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:26.806542 12192 tablet_copy_client.cc:783] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.808537 12192 tablet_copy_client.cc:647] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.818094 12192 tablet_copy_client.cc:523] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.828516  7971 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:26.831182 12114 tablet_copy_client.cc:783] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.831442 12114 tablet_copy_client.cc:647] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.836373 12192 tablet_bootstrap.cc:492] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:26.843938 12139 raft_consensus.cc:1050] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER 03f99420df704238ab300ad6e58d878c to VOTER
I20240503 02:13:26.844842 12139 consensus_queue.cc:237] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:26.851786 12181 tablet_copy_client.cc:783] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.852166 12181 tablet_copy_client.cc:647] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.858335 12181 tablet_copy_client.cc:523] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.865340 12114 tablet_copy_client.cc:523] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.865931 10261 raft_consensus.cc:1203] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:26.871497 12114 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
I20240503 02:13:26.881140  9863 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:26.882094 12138 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:26.890573  8360 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:26.892060 12138 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:26.896373 11952 tablet_replica.cc:330] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: stopping tablet replica
I20240503 02:13:26.897078 11952 raft_consensus.cc:2229] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Raft consensus shutting down.
I20240503 02:13:26.897250 12067 tablet_copy_client.cc:308] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:26.897437 11952 raft_consensus.cc:2258] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Raft consensus is shut down!
I20240503 02:13:26.898108  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet 402de43677c44deab8793da1895d4fb2 from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:42737)
I20240503 02:13:26.898352  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet 402de43677c44deab8793da1895d4fb2 from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:42737: session id = 03f99420df704238ab300ad6e58d878c-402de43677c44deab8793da1895d4fb2
I20240503 02:13:26.900048 11952 ts_tablet_manager.cc:1875] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:26.902776  8382 tablet_copy_source_session.cc:205] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.903589  8099 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:26.904613 12138 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:26.907299 12067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 402de43677c44deab8793da1895d4fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.911038 11952 ts_tablet_manager.cc:1888] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:26.911301 11952 log.cc:1196] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-6/wal/wals/402de43677c44deab8793da1895d4fb2
I20240503 02:13:26.919216  8361 consensus_queue.cc:237] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:26.928247 11952 ts_tablet_manager.cc:902] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:26.933810  9994 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:26.934048  7971 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:26.934558  8231 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:26.934685 11866 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:26.935328 11866 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:26.937317 12067 tablet_copy_client.cc:783] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.937590 12067 tablet_copy_client.cc:647] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.940001 11857 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:26.944826 12067 tablet_copy_client.cc:523] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.944454 11857 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:26.950279 12067 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
I20240503 02:13:26.952718 11952 tablet_copy_client.cc:308] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:26.954414  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet 402de43677c44deab8793da1895d4fb2 from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:43609)
I20240503 02:13:26.954128  7968 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:26.954627  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet 402de43677c44deab8793da1895d4fb2 from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:43609: session id = d826d4dae2aa4e05b2f64c9b17a1126f-402de43677c44deab8793da1895d4fb2
I20240503 02:13:26.957800  8382 tablet_copy_source_session.cc:205] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.959565 11518 raft_consensus.cc:1050] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f to VOTER
I20240503 02:13:26.960712 11518 consensus_queue.cc:237] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:26.961201  9994 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:26.965044  7822 catalog_manager.cc:5484] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:26.971840 11952 ts_tablet_manager.cc:1875] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:13:26.940796 11888 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:26.989558  8100 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:26.996986 10261 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:26.998556  9864 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:26.999771  8100 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:27.003432  7968 consensus_queue.cc:237] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:27.010568 12265 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:27.013515  7823 catalog_manager.cc:5484] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c reported cstate change: config changed from index 4 to 5, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:27.015508 12265 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:27.019170 12006 ts_tablet_manager.cc:902] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:27.050951 11952 ts_tablet_manager.cc:1888] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:13:27.051342 11952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 402de43677c44deab8793da1895d4fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:27.052798 12181 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap starting.
W20240503 02:13:27.060595  8168 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.5:44891, user_credentials={real_user=slave}} blocked reactor thread for 52461.5us
I20240503 02:13:27.062245  8360 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:27.077921  8231 consensus_queue.cc:237] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
W20240503 02:13:27.081238  8298 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.6:40397, user_credentials={real_user=slave}} blocked reactor thread for 98381us
W20240503 02:13:27.081712  8295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.1:37681, user_credentials={real_user=slave}} blocked reactor thread for 101899us
I20240503 02:13:27.082623 10125 raft_consensus.cc:1261] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:27.083577  9863 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:27.085533 10125 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:27.086285 12193 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:27.086411  8099 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:27.087010 12193 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:27.087313 12274 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:13:27.088258  7907 consensus_peers.cc:485] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f2cedba9f9a4e709623fceeffcfd095. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:27.089731 11952 tablet_copy_client.cc:783] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:27.090044 11952 tablet_copy_client.cc:647] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:27.092007 11857 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.097256 12139 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:27.098307  8100 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.104449  9863 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:27.106739  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2448e12efbd44a90b1f8bce9519e73e0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:27.120723  7822 catalog_manager.cc:5484] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:27.125989  8081 tablet_service.cc:1514] Processing DeleteTablet for tablet 2448e12efbd44a90b1f8bce9519e73e0 with delete_type TABLET_DATA_TOMBSTONED (TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54670
I20240503 02:13:27.130040 11952 tablet_copy_client.cc:523] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:27.132143  8101 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:27.130137 12193 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:27.136948 10261 raft_consensus.cc:1203] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240503 02:13:27.128674  8361 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:27.141950  7821 catalog_manager.cc:5484] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 8 to 9, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:27.145898 12264 raft_consensus.cc:1050] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER 03f99420df704238ab300ad6e58d878c to VOTER
I20240503 02:13:27.146317 10261 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.146996 12264 consensus_queue.cc:237] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:27.148011  7821 catalog_manager.cc:5484] T de9c2a3396864696af88be1ca12ee0f1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 8 to 9, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.149096  8361 consensus_queue.cc:237] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:27.155310 12274 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:27.155012  8230 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:27.157931  9994 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:27.158694 11857 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:27.160209  8362 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
W20240503 02:13:27.166492  8298 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c23b06764ce54acb8388de94aea4cd4b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:27.171135  7968 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:27.171340  8230 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:27.174059 11857 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:27.192070 12253 consensus_queue.cc:1035] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
I20240503 02:13:27.192987 12262 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240503 02:13:27.200435 12253 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:27.206351 10124 raft_consensus.cc:2932] T de9c2a3396864696af88be1ca12ee0f1 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.211553  9994 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:27.216569  7971 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:27.217350  7817 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c23b06764ce54acb8388de94aea4cd4b with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:13:27.223361  7821 catalog_manager.cc:5484] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:27.227816 12286 tablet_replica.cc:330] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: stopping tablet replica
I20240503 02:13:27.228061 12286 raft_consensus.cc:2229] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:27.228322 12286 raft_consensus.cc:2258] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:27.231135  8099 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:27.231252 11952 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
I20240503 02:13:27.231835 12260 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:27.234422  8231 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:27.244364 10126 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:27.244518 10125 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:27.246984 12253 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:27.255856 12286 ts_tablet_manager.cc:1875] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:27.274205 12307 ts_tablet_manager.cc:902] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:27.276518  8362 consensus_queue.cc:237] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:27.280057  9994 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:27.280997 12006 tablet_copy_client.cc:308] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:27.281648  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 007b59de142c4b95966494ba097b1a09 from peer ca05c9a8bee148dab8a2a982b8fa2ea4 ({username='slave'} at 127.0.145.8:52015)
I20240503 02:13:27.281839  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 007b59de142c4b95966494ba097b1a09 from peer ca05c9a8bee148dab8a2a982b8fa2ea4 at {username='slave'} at 127.0.145.8:52015: session id = ca05c9a8bee148dab8a2a982b8fa2ea4-007b59de142c4b95966494ba097b1a09
I20240503 02:13:27.283285 12253 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:27.284901  8122 tablet_copy_source_session.cc:205] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:27.286842  7971 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:27.287286 12006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007b59de142c4b95966494ba097b1a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:27.288022 11866 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:27.294649  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c23b06764ce54acb8388de94aea4cd4b with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:27.296345 12260 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:27.298507  9994 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:27.309616 12262 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:27.310171  7968 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:27.315274  8101 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:27.317701 10124 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:27.318562  8100 consensus_queue.cc:237] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:27.322809  7825 catalog_manager.cc:5484] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:27.323657  7823 catalog_manager.cc:5484] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:27.329236  8211 tablet_service.cc:1514] Processing DeleteTablet for tablet c23b06764ce54acb8388de94aea4cd4b with delete_type TABLET_DATA_TOMBSTONED (TS 2942efc0ecb448d2ba2f6bca20fbb29c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35426
I20240503 02:13:27.336512  9863 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:27.338258 11857 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:27.344398 12318 tablet_replica.cc:330] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c: stopping tablet replica
I20240503 02:13:27.345377 12318 raft_consensus.cc:2229] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 02:13:27.345737  8298 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c23b06764ce54acb8388de94aea4cd4b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:27.345753 12318 raft_consensus.cc:2258] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:27.346885 12274 raft_consensus.cc:1050] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f to VOTER
I20240503 02:13:27.347810 12318 ts_tablet_manager.cc:1875] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:27.347703 12274 consensus_queue.cc:237] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:27.368214 12318 ts_tablet_manager.cc:1888] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 02:13:27.368521 12318 log.cc:1196] T c23b06764ce54acb8388de94aea4cd4b P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/c23b06764ce54acb8388de94aea4cd4b
I20240503 02:13:27.370383  7814 catalog_manager.cc:4830] TS 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): tablet c23b06764ce54acb8388de94aea4cd4b (table rebalance_test_table_7 [id=635c8841bff54466a804d93ade171b2f]) successfully deleted
I20240503 02:13:27.371048 12286 ts_tablet_manager.cc:1888] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:27.371271 12286 log.cc:1196] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/2448e12efbd44a90b1f8bce9519e73e0
I20240503 02:13:27.372498  7816 catalog_manager.cc:4830] TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): tablet 2448e12efbd44a90b1f8bce9519e73e0 (table rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]) successfully deleted
I20240503 02:13:27.391553  9862 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:27.399988 10127 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:27.400235  8099 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:27.400355  7968 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:27.400916 11888 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:27.401677 11888 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:27.402288 12139 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:27.409649 12264 raft_consensus.cc:1050] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 to VOTER
I20240503 02:13:27.410578 12264 consensus_queue.cc:237] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:27.421581 12139 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.422341  9861 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:27.423166 12152 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:27.425565  7971 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.426769 10127 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.430125  8101 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.431777  7827 catalog_manager.cc:5484] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.442813 10127 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:27.444435 12152 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:27.445900  7968 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:27.446625 12152 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:27.447456 12307 tablet_copy_client.cc:308] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:27.448266 11952 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:27.448251  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet 6043da7d2e584af885ff6a5bf8f06912 from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:54377)
I20240503 02:13:27.448410  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet 6043da7d2e584af885ff6a5bf8f06912 from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:54377: session id = d826d4dae2aa4e05b2f64c9b17a1126f-6043da7d2e584af885ff6a5bf8f06912
I20240503 02:13:27.448793 11952 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:13:27.449069 11952 ts_tablet_manager.cc:1367] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 0.218s	user 0.039s	sys 0.019s
I20240503 02:13:27.449898 11952 raft_consensus.cc:348] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:27.450203 11952 raft_consensus.cc:729] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:13:27.451839  8252 tablet_copy_source_session.cc:205] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:27.459051  8231 consensus_queue.cc:237] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:27.450496 11952 consensus_queue.cc:260] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:27.462592 10124 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:27.462738  7967 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:27.463508 11952 ts_tablet_manager.cc:1398] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I20240503 02:13:27.464272  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-402de43677c44deab8793da1895d4fb2 received from {username='slave'} at 127.0.145.7:43609
I20240503 02:13:27.464219 11888 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:27.464437  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-402de43677c44deab8793da1895d4fb2 on tablet 402de43677c44deab8793da1895d4fb2 with peer d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:27.464897 11888 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:27.469935 12274 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.471179 10124 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.477360  7828 catalog_manager.cc:5484] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.480401 12151 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.487007 12307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6043da7d2e584af885ff6a5bf8f06912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:27.488180  9861 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.497612 10127 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.498714 12006 tablet_copy_client.cc:783] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:27.499022 12006 tablet_copy_client.cc:647] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:27.498874  7827 catalog_manager.cc:5484] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:27.504492  7967 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.505810  8081 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb5c78a22a74728ab40f1da8f90388c with delete_type TABLET_DATA_TOMBSTONED (TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54670
I20240503 02:13:27.506275 12286 tablet_replica.cc:330] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: stopping tablet replica
I20240503 02:13:27.506479 12286 raft_consensus.cc:2229] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:27.507079 12286 raft_consensus.cc:2258] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:27.507987 12286 ts_tablet_manager.cc:1875] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:27.509215  8100 consensus_queue.cc:237] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
W20240503 02:13:27.514194  8038 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72e2bb75da654e8ab45bba8ac077f28c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:27.517359 12006 tablet_copy_client.cc:523] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:27.518036 12286 ts_tablet_manager.cc:1888] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:27.521348 12286 log.cc:1196] T ecb5c78a22a74728ab40f1da8f90388c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/ecb5c78a22a74728ab40f1da8f90388c
I20240503 02:13:27.522497  7816 catalog_manager.cc:4830] TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): tablet ecb5c78a22a74728ab40f1da8f90388c (table rebalance_test_table_7 [id=635c8841bff54466a804d93ade171b2f]) successfully deleted
I20240503 02:13:27.522778 10261 raft_consensus.cc:1261] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:27.524636  7967 raft_consensus.cc:1261] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:27.525189  8231 raft_consensus.cc:1261] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:27.525722 12260 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:27.526397 12260 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:27.531181 12260 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:27.532508 12354 ts_tablet_manager.cc:902] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512: Initiating tablet copy from peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:27.537873  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ecb5c78a22a74728ab40f1da8f90388c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:27.548269  9861 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:27.549264 12152 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:27.552785 12006 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap starting.
I20240503 02:13:27.554323 10127 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:27.554951 12152 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:27.558049 11957 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:27.565136  7816 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 72e2bb75da654e8ab45bba8ac077f28c with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:13:27.569494 10127 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:27.569571  9861 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:27.576673  7830 catalog_manager.cc:5484] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 9 to 10, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:27.579442  7969 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:27.585217 12152 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:27.587126  7971 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:27.592944 12139 raft_consensus.cc:1050] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 to VOTER
I20240503 02:13:27.593880 12139 consensus_queue.cc:237] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:27.601998 12260 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:27.607390  8231 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:27.613919  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:27.613714  7827 catalog_manager.cc:5484] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:27.617461 10260 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:27.629068  8362 consensus_queue.cc:237] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:27.633416  8231 raft_consensus.cc:1261] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:27.634132 12260 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:27.637789 11866 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:27.642042  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 834aaba965c8443c862e98a070e36cac with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:27.643795  8231 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:27.643900  7830 catalog_manager.cc:5484] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:27.649027 12181 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:27.649525 12181 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap complete.
I20240503 02:13:27.649838 12181 ts_tablet_manager.cc:1367] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent bootstrapping tablet: real 0.597s	user 0.055s	sys 0.005s
I20240503 02:13:27.653198  7971 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:27.657672  7950 tablet_service.cc:1514] Processing DeleteTablet for tablet 834aaba965c8443c862e98a070e36cac with delete_type TABLET_DATA_TOMBSTONED (TS d986fcfb926e47baa7d2d5b99f697334 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34494
I20240503 02:13:27.659798 10261 raft_consensus.cc:1261] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:27.671918  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:27.677153 12354 tablet_copy_client.cc:308] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:37681
I20240503 02:13:27.677903  7991 tablet_copy_service.cc:140] P d986fcfb926e47baa7d2d5b99f697334: Received BeginTabletCopySession request for tablet 4f2cedba9f9a4e709623fceeffcfd095 from peer b85e85e631ab4e04a86bfe5a9f0aa512 ({username='slave'} at 127.0.145.6:46235)
I20240503 02:13:27.678066  7991 tablet_copy_service.cc:161] P d986fcfb926e47baa7d2d5b99f697334: Beginning new tablet copy session on tablet 4f2cedba9f9a4e709623fceeffcfd095 from peer b85e85e631ab4e04a86bfe5a9f0aa512 at {username='slave'} at 127.0.145.6:46235: session id = b85e85e631ab4e04a86bfe5a9f0aa512-4f2cedba9f9a4e709623fceeffcfd095
I20240503 02:13:27.681149  7991 tablet_copy_source_session.cc:205] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 02:13:27.682785 12356 log.cc:924] Time spent T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20240503 02:13:27.691493 12114 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:27.692054 12114 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:27.692983 12264 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:27.695936 12354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f2cedba9f9a4e709623fceeffcfd095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:27.698665 12114 ts_tablet_manager.cc:1367] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.821s	user 0.064s	sys 0.002s
I20240503 02:13:27.699705 12114 raft_consensus.cc:348] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:27.700001 12114 raft_consensus.cc:729] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:27.700225 12114 consensus_queue.cc:260] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:27.710212 12114 ts_tablet_manager.cc:1398] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.000s
I20240503 02:13:27.714651  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-b64f66ace86e42f3ac213ba85bcab584 received from {username='slave'} at 127.0.145.5:48749
I20240503 02:13:27.714818  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session 03f99420df704238ab300ad6e58d878c-b64f66ace86e42f3ac213ba85bcab584 on tablet b64f66ace86e42f3ac213ba85bcab584 with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:27.716431 12371 tablet_replica.cc:330] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: stopping tablet replica
I20240503 02:13:27.717695 12181 raft_consensus.cc:348] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:27.717993 12181 raft_consensus.cc:729] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Initialized, Role: LEARNER
I20240503 02:13:27.718214 12181 consensus_queue.cc:260] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:27.720798 12371 raft_consensus.cc:2229] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:27.724330 12371 pending_rounds.cc:62] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Trying to abort 1 pending ops.
I20240503 02:13:27.725926  8362 consensus_queue.cc:237] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:27.727731 12181 ts_tablet_manager.cc:1398] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent starting tablet: real 0.078s	user 0.002s	sys 0.003s
I20240503 02:13:27.729144  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-0f44b2b53de84a87a8757c0d00bfed37 received from {username='slave'} at 127.0.145.6:56611
I20240503 02:13:27.729279  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-0f44b2b53de84a87a8757c0d00bfed37 on tablet 0f44b2b53de84a87a8757c0d00bfed37 with peer b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:27.730118 12114 ts_tablet_manager.cc:902] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:27.730770 12371 pending_rounds.cc:68] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023421060751613952 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "834aaba965c8443c862e98a070e36cac" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } } }
I20240503 02:13:27.733592 12371 raft_consensus.cc:2894] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 02:13:27.733810 12371 raft_consensus.cc:2866] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:27.734807 12371 raft_consensus.cc:2258] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:27.737520 12354 tablet_copy_client.cc:783] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:27.737825 12354 tablet_copy_client.cc:647] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:27.738508 12371 ts_tablet_manager.cc:1875] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:27.739761 12354 tablet_copy_client.cc:523] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:27.695196 10261 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:27.743623  8100 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:27.744261 11866 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:27.745328 12354 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap starting.
I20240503 02:13:27.747992  9994 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:27.749238  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:27.750892 10260 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 02:13:27.751117  8298 consensus_peers.cc:485] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80b4e6aa20f64473be8bcbf47baf66e4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:27.751718 11888 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:27.751787 12260 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:27.754324 10127 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:27.755131 11888 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:27.797209 12264 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:27.800663  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:27.803014  9995 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:27.805160  7825 catalog_manager.cc:5484] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:27.828963  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:27.839356  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:27.845577 12067 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:27.846151 12067 tablet_bootstrap.cc:492] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:27.846457 12067 ts_tablet_manager.cc:1367] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.896s	user 0.066s	sys 0.038s
I20240503 02:13:27.860564  8362 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:27.861384 12371 ts_tablet_manager.cc:1888] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:27.861601 12371 log.cc:1196] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/834aaba965c8443c862e98a070e36cac
I20240503 02:13:27.864096 12139 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:27.880615 12067 raft_consensus.cc:348] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:27.880949 12067 raft_consensus.cc:729] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:27.881204 12067 consensus_queue.cc:260] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:27.886188  8231 consensus_queue.cc:237] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:27.888903  9861 raft_consensus.cc:1203] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 02:13:27.900143 12067 ts_tablet_manager.cc:1398] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.054s	user 0.003s	sys 0.003s
I20240503 02:13:27.904454  8101 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:27.911698 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:27.918788  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:27.956056  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-402de43677c44deab8793da1895d4fb2 received from {username='slave'} at 127.0.145.5:42737
I20240503 02:13:27.956255  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session 03f99420df704238ab300ad6e58d878c-402de43677c44deab8793da1895d4fb2 on tablet 402de43677c44deab8793da1895d4fb2 with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:27.964514 12139 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:27.970669 10124 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:27.972167 10125 raft_consensus.cc:1203] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:13:27.975255  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:27.979727  7830 catalog_manager.cc:5484] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: config changed from index 9 to 10, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:27.986171 10260 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:27.988184 12006 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:27.988677 12006 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap complete.
I20240503 02:13:27.988991 12006 ts_tablet_manager.cc:1367] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent bootstrapping tablet: real 0.436s	user 0.055s	sys 0.005s
I20240503 02:13:27.989930 12006 raft_consensus.cc:348] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:27.990178 12006 raft_consensus.cc:729] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Initialized, Role: LEARNER
I20240503 02:13:27.997236 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:27.990430 12006 consensus_queue.cc:260] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:28.011905 12006 ts_tablet_manager.cc:1398] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent starting tablet: real 0.023s	user 0.004s	sys 0.000s
I20240503 02:13:28.013442  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-007b59de142c4b95966494ba097b1a09 received from {username='slave'} at 127.0.145.8:52015
I20240503 02:13:28.013581  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-007b59de142c4b95966494ba097b1a09 on tablet 007b59de142c4b95966494ba097b1a09 with peer ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:28.030504 12307 tablet_copy_client.cc:783] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:28.030874 12307 tablet_copy_client.cc:647] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:28.035228  7814 catalog_manager.cc:4830] TS d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): tablet 834aaba965c8443c862e98a070e36cac (table rebalance_test_table_8 [id=8da59f26450d4d25a86bc61be2943c37]) successfully deleted
I20240503 02:13:28.045904 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:28.052901  8230 consensus_queue.cc:237] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:28.054103 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:28.054366  9995 raft_consensus.cc:1203] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12]   Dedup: 1.12->[]
I20240503 02:13:28.056504 10127 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:28.057008 12139 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:28.062817  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:28.062662 12138 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:28.069710 10124 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:28.077410 10260 raft_consensus.cc:1261] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:28.078387 12138 consensus_queue.cc:1035] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:28.080374 10260 raft_consensus.cc:2932] T f14607425da84bb4b162a5789daa844c P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:28.088209 12354 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:28.088783 12354 tablet_bootstrap.cc:492] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap complete.
I20240503 02:13:28.089095 12354 ts_tablet_manager.cc:1367] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent bootstrapping tablet: real 0.344s	user 0.050s	sys 0.016s
I20240503 02:13:28.090118 12354 raft_consensus.cc:348] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:28.090418 12354 raft_consensus.cc:729] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Initialized, Role: LEARNER
I20240503 02:13:28.098678 12354 consensus_queue.cc:260] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:28.099435 12354 ts_tablet_manager.cc:1398] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I20240503 02:13:28.100647  7991 tablet_copy_service.cc:342] P d986fcfb926e47baa7d2d5b99f697334: Request end of tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-4f2cedba9f9a4e709623fceeffcfd095 received from {username='slave'} at 127.0.145.6:46235
I20240503 02:13:28.101322  7991 tablet_copy_service.cc:433] P d986fcfb926e47baa7d2d5b99f697334: ending tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-4f2cedba9f9a4e709623fceeffcfd095 on tablet 4f2cedba9f9a4e709623fceeffcfd095 with peer b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:28.102540  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:28.104111 12114 tablet_copy_client.cc:308] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:28.115284  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:28.116413 12307 tablet_copy_client.cc:523] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:28.119634 12354 ts_tablet_manager.cc:902] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:28.124887  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet c23b06764ce54acb8388de94aea4cd4b from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:42737)
I20240503 02:13:28.125123  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet c23b06764ce54acb8388de94aea4cd4b from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:42737: session id = 03f99420df704238ab300ad6e58d878c-c23b06764ce54acb8388de94aea4cd4b
I20240503 02:13:28.126031  7825 catalog_manager.cc:5484] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 10 to 11, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:28.128239  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f14607425da84bb4b162a5789daa844c with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:28.129112  8382 tablet_copy_source_session.cc:205] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:28.130213  8341 tablet_service.cc:1514] Processing DeleteTablet for tablet f14607425da84bb4b162a5789daa844c with delete_type TABLET_DATA_TOMBSTONED (TS 8d7c037d50c241b484ee843b06313a64 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:37080
I20240503 02:13:28.138254  7971 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:28.138473  8101 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:28.139142 12138 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:28.139720 12139 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:28.142554  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:13:28.146668  8167 consensus_peers.cc:485] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d69c71d9dfd47a1a2429e7caf5e33ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:28.160838 12307 tablet_bootstrap.cc:492] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
I20240503 02:13:28.167510 12067 ts_tablet_manager.cc:902] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:28.168658 10261 raft_consensus.cc:1203] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:28.173209 12114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23b06764ce54acb8388de94aea4cd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.184293  8100 consensus_queue.cc:237] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
W20240503 02:13:28.187367  8038 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72e2bb75da654e8ab45bba8ac077f28c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:28.187381  9861 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:28.188154 12151 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:28.191220 12151 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:28.196192  9861 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:28.196595 12138 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:28.205039  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 72e2bb75da654e8ab45bba8ac077f28c with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:28.206243  7967 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:28.214987  7828 catalog_manager.cc:5484] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:28.235287 12413 tablet_replica.cc:330] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64: stopping tablet replica
I20240503 02:13:28.236095 12067 tablet_copy_client.cc:308] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:28.236550 12413 raft_consensus.cc:2229] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:28.237529  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet 80b4e6aa20f64473be8bcbf47baf66e4 from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:42737)
I20240503 02:13:28.239696  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet 80b4e6aa20f64473be8bcbf47baf66e4 from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:42737: session id = 03f99420df704238ab300ad6e58d878c-80b4e6aa20f64473be8bcbf47baf66e4
I20240503 02:13:28.239904  8100 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:28.241489  9861 raft_consensus.cc:1203] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:13:28.243104  8382 tablet_copy_source_session.cc:205] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:28.246520 12067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b4e6aa20f64473be8bcbf47baf66e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.252899 12413 pending_rounds.cc:62] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64: Trying to abort 1 pending ops.
I20240503 02:13:28.253206 12413 pending_rounds.cc:68] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64: Aborting op as it isn't in flight: id { term: 1 index: 10 } timestamp: 7023421061502271488 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f14607425da84bb4b162a5789daa844c" old_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } } new_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } } }
I20240503 02:13:28.254683  7829 catalog_manager.cc:5484] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c reported cstate change: config changed from index 10 to 11, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:28.254494 12413 raft_consensus.cc:2894] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Aborting config change with OpId 1.10: Aborted: Op aborted
I20240503 02:13:28.256062 12413 raft_consensus.cc:2866] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:28.256598 12413 raft_consensus.cc:2258] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:28.258689 12413 ts_tablet_manager.cc:1875] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:28.270555 10124 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:28.271351 12151 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:28.286835  7950 tablet_service.cc:1514] Processing DeleteTablet for tablet 72e2bb75da654e8ab45bba8ac077f28c with delete_type TABLET_DATA_TOMBSTONED (TS d986fcfb926e47baa7d2d5b99f697334 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:34494
I20240503 02:13:28.287339 12371 tablet_replica.cc:330] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334: stopping tablet replica
I20240503 02:13:28.287544 12371 raft_consensus.cc:2229] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:28.287794 12371 raft_consensus.cc:2258] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:28.287528 10124 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:28.289007 12371 ts_tablet_manager.cc:1875] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:28.307978 12371 ts_tablet_manager.cc:1888] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 02:13:28.318728 12371 log.cc:1196] T 72e2bb75da654e8ab45bba8ac077f28c P d986fcfb926e47baa7d2d5b99f697334: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/72e2bb75da654e8ab45bba8ac077f28c
I20240503 02:13:28.320021  7814 catalog_manager.cc:4830] TS d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): tablet 72e2bb75da654e8ab45bba8ac077f28c (table rebalance_test_table_8 [id=8da59f26450d4d25a86bc61be2943c37]) successfully deleted
I20240503 02:13:28.337617 12114 tablet_copy_client.cc:783] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:28.342440 12114 tablet_copy_client.cc:647] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:28.351842 12114 tablet_copy_client.cc:523] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:28.364589 12114 tablet_bootstrap.cc:492] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
I20240503 02:13:28.367817 12067 tablet_copy_client.cc:783] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:28.372341 12067 tablet_copy_client.cc:647] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:28.374097  7971 consensus_queue.cc:237] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:28.375857 12067 tablet_copy_client.cc:523] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:28.391588 12067 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
W20240503 02:13:28.407114  7906 consensus_peers.cc:485] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ea56405515a40f785726bd58ec3d26a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:28.408288  9995 raft_consensus.cc:1261] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:28.409251 12408 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:28.414194 11866 raft_consensus.cc:1050] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f to VOTER
I20240503 02:13:28.414461 12354 tablet_copy_client.cc:308] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:28.415766  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 72e2bb75da654e8ab45bba8ac077f28c from peer b85e85e631ab4e04a86bfe5a9f0aa512 ({username='slave'} at 127.0.145.6:56611)
I20240503 02:13:28.415942  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 72e2bb75da654e8ab45bba8ac077f28c from peer b85e85e631ab4e04a86bfe5a9f0aa512 at {username='slave'} at 127.0.145.6:56611: session id = b85e85e631ab4e04a86bfe5a9f0aa512-72e2bb75da654e8ab45bba8ac077f28c
I20240503 02:13:28.415833 11866 consensus_queue.cc:237] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:28.417263  8362 raft_consensus.cc:1261] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:28.419229  8122 tablet_copy_source_session.cc:205] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:28.419081 12411 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:28.421926 12354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e2bb75da654e8ab45bba8ac077f28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.424192 12192 tablet_bootstrap.cc:492] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:28.432614  9861 raft_consensus.cc:1261] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:28.433357 12264 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:28.436471 10125 raft_consensus.cc:1261] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:28.437104  8231 raft_consensus.cc:1261] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:28.438445  8100 raft_consensus.cc:1261] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:28.439237 12264 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:28.439397 12413 ts_tablet_manager.cc:1888] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 02:13:28.439721 12413 log.cc:1196] T f14607425da84bb4b162a5789daa844c P 8d7c037d50c241b484ee843b06313a64: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/wals/f14607425da84bb4b162a5789daa844c
I20240503 02:13:28.439735 12264 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:28.442459  9994 raft_consensus.cc:1203] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240503 02:13:28.445863 12264 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:28.448009  7817 catalog_manager.cc:4830] TS 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): tablet f14607425da84bb4b162a5789daa844c (table rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]) successfully deleted
I20240503 02:13:28.450583  8362 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:28.444854 12411 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:28.453910 12354 tablet_copy_client.cc:783] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:28.454159 12354 tablet_copy_client.cc:647] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:28.456578 12354 tablet_copy_client.cc:523] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:28.458475 12192 tablet_bootstrap.cc:492] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334: Bootstrap complete.
I20240503 02:13:28.458982 12192 ts_tablet_manager.cc:1367] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 1.623s	user 0.073s	sys 0.001s
I20240503 02:13:28.459851 12192 raft_consensus.cc:348] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } }
I20240503 02:13:28.460258 12192 raft_consensus.cc:729] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: LEARNER
I20240503 02:13:28.460573 12192 consensus_queue.cc:260] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } }
I20240503 02:13:28.461340 12192 ts_tablet_manager.cc:1398] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:13:28.460129  7830 catalog_manager.cc:5484] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 6 to 7, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:28.462414  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session d986fcfb926e47baa7d2d5b99f697334-7e966caa651e4e42890be5d79f851a1f received from {username='slave'} at 127.0.145.1:39931
I20240503 02:13:28.462584  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session d986fcfb926e47baa7d2d5b99f697334-7e966caa651e4e42890be5d79f851a1f on tablet 7e966caa651e4e42890be5d79f851a1f with peer d986fcfb926e47baa7d2d5b99f697334
I20240503 02:13:28.479295 12354 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap starting.
I20240503 02:13:28.481093 11957 raft_consensus.cc:1050] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 to VOTER
I20240503 02:13:28.482110 11957 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:28.499164 12264 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:28.500684  8101 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:28.507016  7830 catalog_manager.cc:5484] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:28.511418  8230 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:28.512576 12152 raft_consensus.cc:1050] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER 03f99420df704238ab300ad6e58d878c to VOTER
I20240503 02:13:28.517067 12152 consensus_queue.cc:237] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:28.544138 10125 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:28.556684  9994 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:13:28.557408 12151 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:13:28.567587  9996 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:28.581349 10261 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:13:28.582167 12151 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:13:28.586545  9862 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:13:28.592804 12151 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20240503 02:13:28.593626  8362 consensus_queue.cc:237] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:28.596539  8100 raft_consensus.cc:1261] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:28.596266 12151 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:28.597427 12264 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:13:28.601004  8295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.3:38203, user_credentials={real_user=slave}} blocked reactor thread for 75989.3us
I20240503 02:13:28.608007 10261 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:28.619027  9995 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:28.619787  8231 raft_consensus.cc:1261] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:28.620533 10125 raft_consensus.cc:1261] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:28.621587 11866 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:28.623925 12264 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:28.624359  7828 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 12 to 13, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:28.630730  8231 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:28.639703  7830 catalog_manager.cc:5484] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:28.640683 10261 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:28.642791  8362 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:28.643463 12152 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:28.645082 12260 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:28.645459 12152 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.003s
I20240503 02:13:28.651331  7971 raft_consensus.cc:1203] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 02:13:28.659363 12477 ts_tablet_manager.cc:902] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:28.673111 11957 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:28.680312 10261 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:28.681208  9860 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:28.681617  9844 tablet_service.cc:1514] Processing DeleteTablet for tablet 402de43677c44deab8793da1895d4fb2 with delete_type TABLET_DATA_TOMBSTONED (TS 03f99420df704238ab300ad6e58d878c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35050
I20240503 02:13:28.682500 12471 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240503 02:13:28.700577  8100 consensus_queue.cc:237] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:28.702445 12477 tablet_copy_client.cc:308] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:28.703507  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac from peer ca05c9a8bee148dab8a2a982b8fa2ea4 ({username='slave'} at 127.0.145.8:44853)
I20240503 02:13:28.703748  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet 2d69c71d9dfd47a1a2429e7caf5e33ac from peer ca05c9a8bee148dab8a2a982b8fa2ea4 at {username='slave'} at 127.0.145.8:44853: session id = ca05c9a8bee148dab8a2a982b8fa2ea4-2d69c71d9dfd47a1a2429e7caf5e33ac
I20240503 02:13:28.707620  8252 tablet_copy_source_session.cc:205] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:28.708703  8101 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:28.709753 10125 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:28.713305 12477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.714907  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 402de43677c44deab8793da1895d4fb2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:28.717628  8230 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:28.717989  7967 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:28.718681 12472 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:28.721580 12471 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:28.727285  9995 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:28.729102 12152 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:28.733366 12483 tablet_replica.cc:330] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: stopping tablet replica
I20240503 02:13:28.733671 12483 raft_consensus.cc:2229] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Raft consensus shutting down.
I20240503 02:13:28.733881 12483 pending_rounds.cc:62] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: Trying to abort 1 pending ops.
I20240503 02:13:28.740335  9860 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:28.746304 12477 tablet_copy_client.cc:783] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:28.746598 12477 tablet_copy_client.cc:647] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:28.748909 12151 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:28.752409  9994 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:28.754406 12477 tablet_copy_client.cc:523] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:28.756963  8100 consensus_queue.cc:237] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:28.760756  7830 catalog_manager.cc:5484] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:28.784811 12354 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:28.785454 12354 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap complete.
I20240503 02:13:28.785764 12354 ts_tablet_manager.cc:1367] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent bootstrapping tablet: real 0.307s	user 0.052s	sys 0.015s
I20240503 02:13:28.787535  8362 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:28.788606 12477 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap starting.
I20240503 02:13:28.788255  8231 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:28.793795  7825 catalog_manager.cc:5484] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:28.809895  8231 raft_consensus.cc:1261] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:28.811115  7971 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:28.818718  8362 consensus_queue.cc:237] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
W20240503 02:13:28.820631  8035 consensus_peers.cc:485] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a8affdde4d8417f95f0585dd01e7d59. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:28.823261 12152 consensus_queue.cc:1035] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:28.824502  9859 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:28.825826  7967 raft_consensus.cc:1261] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:28.826413 12472 consensus_queue.cc:1035] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:28.829175 12472 raft_consensus.cc:1050] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 to VOTER
I20240503 02:13:28.829690  8101 raft_consensus.cc:1261] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:28.830057 12472 consensus_queue.cc:237] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:28.834028 12458 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:28.734017 12483 pending_rounds.cc:68] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023421064866398208 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "402de43677c44deab8793da1895d4fb2" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } } }
I20240503 02:13:28.842258 12483 raft_consensus.cc:2894] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240503 02:13:28.842514 12483 raft_consensus.cc:2866] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:28.842793 12483 raft_consensus.cc:2258] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Raft consensus is shut down!
I20240503 02:13:28.845409 10124 raft_consensus.cc:1261] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:28.847203 12354 raft_consensus.cc:348] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:28.849318 12506 raft_consensus.cc:2932] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:28.853081 12456 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:28.855325  8231 raft_consensus.cc:2932] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:28.857108 11866 consensus_queue.cc:1035] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:28.865422  8101 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:28.868551  8100 consensus_queue.cc:237] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:28.870115  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 402de43677c44deab8793da1895d4fb2 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:28.871840  7830 catalog_manager.cc:5484] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:28.877805  8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 402de43677c44deab8793da1895d4fb2 with delete_type TABLET_DATA_TOMBSTONED (TS 2942efc0ecb448d2ba2f6bca20fbb29c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35426
I20240503 02:13:28.882735  8231 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:28.883970 10261 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:28.884498  8231 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:28.884792  7971 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:28.885514 12507 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:28.885682 12505 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:28.886490 12505 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:28.887652  7830 catalog_manager.cc:5484] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:28.906536  9995 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:28.906960 12505 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:28.909224 12483 ts_tablet_manager.cc:1875] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:28.910765 12505 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:28.916301 12354 raft_consensus.cc:729] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Initialized, Role: LEARNER
I20240503 02:13:28.917397 12354 consensus_queue.cc:260] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:28.926328 12506 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:28.928454  8230 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:28.928875 12354 ts_tablet_manager.cc:1398] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent starting tablet: real 0.143s	user 0.003s	sys 0.000s
I20240503 02:13:28.936208  7827 catalog_manager.cc:5484] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 6 to 7, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:28.940694  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4af5884849ec4ac3891ebf0f08ee2cff with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:28.942122  7950 tablet_service.cc:1514] Processing DeleteTablet for tablet 4af5884849ec4ac3891ebf0f08ee2cff with delete_type TABLET_DATA_TOMBSTONED (TS d986fcfb926e47baa7d2d5b99f697334 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34494
I20240503 02:13:28.950218  8101 consensus_queue.cc:237] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:28.952826 12506 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:28.960415  8230 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:28.960949 12516 tablet_replica.cc:330] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c: stopping tablet replica
I20240503 02:13:28.961191 12505 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:28.961345 10261 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:28.932433 10125 raft_consensus.cc:2932] T 402de43677c44deab8793da1895d4fb2 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:28.965483  7967 raft_consensus.cc:2932] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:13:28.968475  7825 catalog_manager.cc:5484] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:28.969466 12516 raft_consensus.cc:2229] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:28.970101 12516 raft_consensus.cc:2258] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:28.970336 12494 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:28.971585 12516 ts_tablet_manager.cc:1875] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:28.974946  7816 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4af5884849ec4ac3891ebf0f08ee2cff with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:13:28.977026  9995 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:13:28.979494  8038 consensus_peers.cc:485] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4af5884849ec4ac3891ebf0f08ee2cff. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:28.983989 12523 tablet_replica.cc:330] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: stopping tablet replica
I20240503 02:13:28.983573  9994 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:28.984285  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-72e2bb75da654e8ab45bba8ac077f28c received from {username='slave'} at 127.0.145.6:56611
I20240503 02:13:28.984490  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-72e2bb75da654e8ab45bba8ac077f28c on tablet 72e2bb75da654e8ab45bba8ac077f28c with peer b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:28.985987 12523 raft_consensus.cc:2229] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:28.987422 12523 raft_consensus.cc:2258] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:28.989781  8101 consensus_queue.cc:237] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:28.990777 12505 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:28.992580 12523 ts_tablet_manager.cc:1875] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:28.994184 10259 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:28.995263 12472 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:28.996455  7971 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:28.993052  8231 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.002425  8230 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:29.006430  7969 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:29.008141 12525 ts_tablet_manager.cc:902] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:29.010077  7830 catalog_manager.cc:5484] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 7 to 8, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:29.012228  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 007b59de142c4b95966494ba097b1a09. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.012712 12151 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:29.013294 12505 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:29.013177  9995 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.019232  8230 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:29.027532 12151 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.031082 12523 ts_tablet_manager.cc:1888] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:13:29.032258 12523 log.cc:1196] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/4af5884849ec4ac3891ebf0f08ee2cff
I20240503 02:13:29.033313  7814 catalog_manager.cc:4830] TS d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): tablet 4af5884849ec4ac3891ebf0f08ee2cff (table rebalance_test_table_8 [id=8da59f26450d4d25a86bc61be2943c37]) successfully deleted
I20240503 02:13:29.034296  7971 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.033633 10260 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.041224  8230 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.071421 12411 raft_consensus.cc:1050] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334: attempting to promote NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 to VOTER
I20240503 02:13:29.072288 12411 consensus_queue.cc:237] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:29.081532  8101 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:29.082334 12408 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:29.082660  9995 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:29.083372 12408 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:29.088858 10125 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:29.091189 12408 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:29.106128  7829 catalog_manager.cc:5484] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 5 to 6, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.111686 12410 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:29.124794  8100 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:29.125770  9995 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:29.127686  7828 catalog_manager.cc:5484] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 9 to 10, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:29.130625 12516 ts_tablet_manager.cc:1888] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 02:13:29.130872 12516 log.cc:1196] T 402de43677c44deab8793da1895d4fb2 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/402de43677c44deab8793da1895d4fb2
I20240503 02:13:29.133188  7814 catalog_manager.cc:4830] TS 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): tablet 402de43677c44deab8793da1895d4fb2 (table rebalance_test_table_3 [id=9af1e1dd28c144d9b52b73387f91d232]) successfully deleted
I20240503 02:13:29.134056 12483 ts_tablet_manager.cc:1888] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:13:29.142757 12483 log.cc:1196] T 402de43677c44deab8793da1895d4fb2 P 03f99420df704238ab300ad6e58d878c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/wals/402de43677c44deab8793da1895d4fb2
I20240503 02:13:29.157300  7817 catalog_manager.cc:4830] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): tablet 402de43677c44deab8793da1895d4fb2 (table rebalance_test_table_3 [id=9af1e1dd28c144d9b52b73387f91d232]) successfully deleted
I20240503 02:13:29.175309  8100 consensus_queue.cc:237] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:29.178866 10260 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:29.179585 12506 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:29.180233  7971 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:29.181000 12506 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:29.182863 12506 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.184705 10261 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.185186  7969 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.193003  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 007b59de142c4b95966494ba097b1a09 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:29.194976  7825 catalog_manager.cc:5484] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 6 to 7, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.199563  8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 007b59de142c4b95966494ba097b1a09 with delete_type TABLET_DATA_TOMBSTONED (TS 2942efc0ecb448d2ba2f6bca20fbb29c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35426
I20240503 02:13:29.202893 12516 tablet_replica.cc:330] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: stopping tablet replica
I20240503 02:13:29.203233 12516 raft_consensus.cc:2229] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:29.203801 12516 raft_consensus.cc:2258] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:29.205221 12516 ts_tablet_manager.cc:1875] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:29.230118 12067 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:29.230582 12067 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:29.230916 12067 ts_tablet_manager.cc:1367] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.839s	user 0.049s	sys 0.018s
I20240503 02:13:29.231688 12067 raft_consensus.cc:348] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:29.232012 12067 raft_consensus.cc:729] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:29.232244 12067 consensus_queue.cc:260] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:29.232820 12067 ts_tablet_manager.cc:1398] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:13:29.233507  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-80b4e6aa20f64473be8bcbf47baf66e4 received from {username='slave'} at 127.0.145.5:42737
I20240503 02:13:29.233652  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session 03f99420df704238ab300ad6e58d878c-80b4e6aa20f64473be8bcbf47baf66e4 on tablet 80b4e6aa20f64473be8bcbf47baf66e4 with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:29.235723 12516 ts_tablet_manager.cc:1888] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:13:29.235864 12516 log.cc:1196] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/007b59de142c4b95966494ba097b1a09
I20240503 02:13:29.237715  7814 catalog_manager.cc:4830] TS 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): tablet 007b59de142c4b95966494ba097b1a09 (table rebalance_test_table_3 [id=9af1e1dd28c144d9b52b73387f91d232]) successfully deleted
I20240503 02:13:29.252991 12477 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:29.253538 12477 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap complete.
I20240503 02:13:29.257162 12477 ts_tablet_manager.cc:1367] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent bootstrapping tablet: real 0.469s	user 0.058s	sys 0.017s
I20240503 02:13:29.260851 12114 tablet_bootstrap.cc:492] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:29.261451 12114 tablet_bootstrap.cc:492] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:29.261760 12114 ts_tablet_manager.cc:1367] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.900s	user 0.051s	sys 0.014s
I20240503 02:13:29.262420 12477 raft_consensus.cc:348] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:29.262785 12477 raft_consensus.cc:729] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Initialized, Role: LEARNER
I20240503 02:13:29.262706 12114 raft_consensus.cc:348] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:29.263000 12114 raft_consensus.cc:729] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:29.263036 12477 consensus_queue.cc:260] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:29.263224 12114 consensus_queue.cc:260] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:29.263795 12477 ts_tablet_manager.cc:1398] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.002s
I20240503 02:13:29.263795 12114 ts_tablet_manager.cc:1398] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:13:29.264705  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-c23b06764ce54acb8388de94aea4cd4b received from {username='slave'} at 127.0.145.5:42737
I20240503 02:13:29.264873  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session 03f99420df704238ab300ad6e58d878c-c23b06764ce54acb8388de94aea4cd4b on tablet c23b06764ce54acb8388de94aea4cd4b with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:29.266314  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-2d69c71d9dfd47a1a2429e7caf5e33ac received from {username='slave'} at 127.0.145.8:44853
I20240503 02:13:29.266471  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-2d69c71d9dfd47a1a2429e7caf5e33ac on tablet 2d69c71d9dfd47a1a2429e7caf5e33ac with peer ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:29.268431  9859 raft_consensus.cc:1203] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:13:29.269239  9859 raft_consensus.cc:1203] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
W20240503 02:13:29.283282  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 007b59de142c4b95966494ba097b1a09. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.288295 12307 tablet_bootstrap.cc:492] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:29.291224 12307 tablet_bootstrap.cc:492] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:13:29.292145 12307 ts_tablet_manager.cc:1367] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 1.131s	user 0.047s	sys 0.010s
I20240503 02:13:29.293418 12307 raft_consensus.cc:348] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:29.295301 12307 raft_consensus.cc:729] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:13:29.295883 12307 consensus_queue.cc:260] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:29.297308 12307 ts_tablet_manager.cc:1398] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240503 02:13:29.298869  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-6043da7d2e584af885ff6a5bf8f06912 received from {username='slave'} at 127.0.145.7:54377
I20240503 02:13:29.299032  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-6043da7d2e584af885ff6a5bf8f06912 on tablet 6043da7d2e584af885ff6a5bf8f06912 with peer d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:29.299103 10124 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:29.303395 12525 tablet_copy_client.cc:308] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:37681
I20240503 02:13:29.307132  7991 tablet_copy_service.cc:140] P d986fcfb926e47baa7d2d5b99f697334: Received BeginTabletCopySession request for tablet 5ea56405515a40f785726bd58ec3d26a from peer ca05c9a8bee148dab8a2a982b8fa2ea4 ({username='slave'} at 127.0.145.8:52797)
I20240503 02:13:29.307366  7991 tablet_copy_service.cc:161] P d986fcfb926e47baa7d2d5b99f697334: Beginning new tablet copy session on tablet 5ea56405515a40f785726bd58ec3d26a from peer ca05c9a8bee148dab8a2a982b8fa2ea4 at {username='slave'} at 127.0.145.8:52797: session id = ca05c9a8bee148dab8a2a982b8fa2ea4-5ea56405515a40f785726bd58ec3d26a
I20240503 02:13:29.310343  8362 consensus_queue.cc:237] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:29.315050  7991 tablet_copy_source_session.cc:205] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:29.316967 12525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea56405515a40f785726bd58ec3d26a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.320979  7969 consensus_queue.cc:237] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:29.327554  9995 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:29.328490 12410 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:29.334102 12525 tablet_copy_client.cc:783] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:29.334388 12525 tablet_copy_client.cc:647] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:29.337991 12307 ts_tablet_manager.cc:902] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:29.341805  9995 raft_consensus.cc:1203] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
W20240503 02:13:29.346256  8298 consensus_peers.cc:485] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3d4b912095244de816d3089a31b8649. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.347625 10126 raft_consensus.cc:1261] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:29.348131  8230 raft_consensus.cc:1261] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:29.348222 12264 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:29.348243  9995 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:29.348868 12264 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:29.346064 12410 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:29.363699 12525 tablet_copy_client.cc:523] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:29.368325 12114 ts_tablet_manager.cc:902] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:29.370481 12525 tablet_bootstrap.cc:492] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap starting.
I20240503 02:13:29.375694 12260 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:29.379118  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4f2cedba9f9a4e709623fceeffcfd095 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:29.379613  7825 catalog_manager.cc:5484] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 10 to 11, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:29.383674 10126 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:29.384197  8081 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f2cedba9f9a4e709623fceeffcfd095 with delete_type TABLET_DATA_TOMBSTONED (TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:54670
I20240503 02:13:29.387617 10127 raft_consensus.cc:1261] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:29.388482 12408 consensus_queue.cc:1035] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:29.390795  8230 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:29.390539 10127 raft_consensus.cc:2932] T 4f2cedba9f9a4e709623fceeffcfd095 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:29.396631 12575 tablet_replica.cc:330] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: stopping tablet replica
I20240503 02:13:29.396935 12575 raft_consensus.cc:2229] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:29.401654 12575 raft_consensus.cc:2258] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:29.404924  7821 catalog_manager.cc:5484] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 6 to 7, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:29.403254 12575 ts_tablet_manager.cc:1875] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:29.420811 12138 raft_consensus.cc:1050] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER d986fcfb926e47baa7d2d5b99f697334 to VOTER
I20240503 02:13:29.422379 12138 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } }
I20240503 02:13:29.433974 12307 tablet_copy_client.cc:308] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:29.435184  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 8a8affdde4d8417f95f0585dd01e7d59 from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:59845)
I20240503 02:13:29.435405  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 8a8affdde4d8417f95f0585dd01e7d59 from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:59845: session id = d826d4dae2aa4e05b2f64c9b17a1126f-8a8affdde4d8417f95f0585dd01e7d59
I20240503 02:13:29.436393 12575 ts_tablet_manager.cc:1888] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 02:13:29.436539 12575 log.cc:1196] T 4f2cedba9f9a4e709623fceeffcfd095 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/4f2cedba9f9a4e709623fceeffcfd095
I20240503 02:13:29.436693  8230 consensus_queue.cc:237] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:29.440222  8122 tablet_copy_source_session.cc:205] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:29.443795 12307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8affdde4d8417f95f0585dd01e7d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.452113  8362 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:29.452884 11888 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:29.442940  7816 catalog_manager.cc:4830] TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): tablet 4f2cedba9f9a4e709623fceeffcfd095 (table rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]) successfully deleted
I20240503 02:13:29.459420  7967 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:29.460429 12138 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:13:29.466161  8168 consensus_peers.cc:485] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d69c71d9dfd47a1a2429e7caf5e33ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.467146  8100 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:29.470191 12139 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:29.476038 12307 tablet_copy_client.cc:783] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:29.477002 12307 tablet_copy_client.cc:647] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:29.481000 12307 tablet_copy_client.cc:523] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:29.489943 10261 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:29.491343 10261 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:29.492065 12139 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:29.492825 12139 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:29.493534 12307 tablet_bootstrap.cc:492] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
I20240503 02:13:29.506201  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:29.512491  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:29.515365 12139 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:29.525226  8361 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:29.536713  7828 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 8 to 9, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:29.546439 12139 raft_consensus.cc:1013] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: attempt to promote peer ca05c9a8bee148dab8a2a982b8fa2ea4: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 02:13:29.547654 10260 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:29.559772 12114 tablet_copy_client.cc:308] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:29.560595  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 4af5884849ec4ac3891ebf0f08ee2cff from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:48749)
I20240503 02:13:29.560844  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 4af5884849ec4ac3891ebf0f08ee2cff from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:48749: session id = 03f99420df704238ab300ad6e58d878c-4af5884849ec4ac3891ebf0f08ee2cff
I20240503 02:13:29.564571  7967 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:29.566041  8122 tablet_copy_source_session.cc:205] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:29.570225 12114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af5884849ec4ac3891ebf0f08ee2cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.570415 12264 raft_consensus.cc:1050] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER 03f99420df704238ab300ad6e58d878c to VOTER
I20240503 02:13:29.571420 12264 consensus_queue.cc:237] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:29.592126  8101 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.592407 12114 tablet_copy_client.cc:783] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:29.593020 12114 tablet_copy_client.cc:647] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:29.598134 12114 tablet_copy_client.cc:523] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:29.599418  7821 catalog_manager.cc:5484] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 4 to 5, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.606504 12114 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
I20240503 02:13:29.607096  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:29.614594  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:29.640604  9859 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:29.640882  9995 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:29.641279  8101 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:29.641690 12260 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:29.642383 12260 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:29.642997 12458 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:29.647643 11888 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.659116  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:29.662039 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:29.694778  7969 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:29.700971 10260 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.706324 12604 raft_consensus.cc:1050] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 to VOTER
I20240503 02:13:29.707234 12604 consensus_queue.cc:237] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:29.709780 12458 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:29.710988  8100 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:13:29.711517  8168 consensus_peers.cc:485] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d69c71d9dfd47a1a2429e7caf5e33ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.711753 12139 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:29.715118  9995 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:29.720999 10124 raft_consensus.cc:1203] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:29.722868  7969 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:29.725234 11888 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:29.727519  9860 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:29.731235  7969 raft_consensus.cc:858] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }.
  Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }: 
  New config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
W20240503 02:13:29.731379  7969 raft_consensus.cc:1565] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }.
  Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }: 
  New config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:29.733671 12067 ts_tablet_manager.cc:902] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:29.734102 12260 raft_consensus.cc:1050] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER 03f99420df704238ab300ad6e58d878c to VOTER
I20240503 02:13:29.735198 12260 consensus_queue.cc:237] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:29.726971  8100 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:29.744874 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:29.744932 10260 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:29.760167 12628 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:29.763355 12138 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.769776 10260 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.772755  7967 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:29.773550 12260 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:29.777386  9995 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:29.779090 12458 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:29.780726  7821 catalog_manager.cc:5484] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:29.790203  9860 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:29.791003 12264 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240503 02:13:29.791442  8362 consensus_queue.cc:237] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:29.793256  7821 catalog_manager.cc:5484] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 5 to 6, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:29.796952  8100 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.801849  9859 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:29.803208  9994 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:29.805025 12504 raft_consensus.cc:1050] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 to VOTER
I20240503 02:13:29.805922 12504 consensus_queue.cc:237] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:29.808133  7969 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.810871  9994 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:13:29.811506 12506 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:13:29.813653  9859 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:13:29.814280 12506 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:13:29.824527 10124 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:13:29.825246 12506 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:13:29.846038 12260 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:29.846983 12260 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:29.851244  7971 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.853650 12264 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:29.855216  9862 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:29.856215  9995 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:29.861281  7828 catalog_manager.cc:5484] T 80b4e6aa20f64473be8bcbf47baf66e4 P 03f99420df704238ab300ad6e58d878c reported cstate change: config changed from index 8 to 9, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:29.865665  8081 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b4e6aa20f64473be8bcbf47baf66e4 with delete_type TABLET_DATA_TOMBSTONED (TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:54670
I20240503 02:13:29.866083 12575 tablet_replica.cc:330] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: stopping tablet replica
I20240503 02:13:29.866297 12575 raft_consensus.cc:2229] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:29.866683 12575 raft_consensus.cc:2258] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:29.868046 12575 ts_tablet_manager.cc:1875] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:29.869535 12307 tablet_bootstrap.cc:492] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:29.869969 12307 tablet_bootstrap.cc:492] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:13:29.870235 12307 ts_tablet_manager.cc:1367] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 0.377s	user 0.044s	sys 0.012s
I20240503 02:13:29.870592 12458 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:29.871095 12307 raft_consensus.cc:348] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:29.870872  7968 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:29.871342 12307 raft_consensus.cc:729] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:13:29.871616 12307 consensus_queue.cc:260] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:13:29.873266 12307 ts_tablet_manager.cc:1398] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:13:29.876816  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-8a8affdde4d8417f95f0585dd01e7d59 received from {username='slave'} at 127.0.145.7:59845
I20240503 02:13:29.876961  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-8a8affdde4d8417f95f0585dd01e7d59 on tablet 8a8affdde4d8417f95f0585dd01e7d59 with peer d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:29.877811 12575 ts_tablet_manager.cc:1888] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:29.878242 12575 log.cc:1196] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/80b4e6aa20f64473be8bcbf47baf66e4
I20240503 02:13:29.879459  7816 catalog_manager.cc:4830] TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): tablet 80b4e6aa20f64473be8bcbf47baf66e4 (table rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]) successfully deleted
I20240503 02:13:29.879573  7828 catalog_manager.cc:5484] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:29.882179  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:29.885141  8231 consensus_queue.cc:237] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
W20240503 02:13:29.888710  8168 consensus_peers.cc:485] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d69c71d9dfd47a1a2429e7caf5e33ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.892535 10260 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:29.894665 12604 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:29.897392 12592 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.898147 12506 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:29.900632  9996 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:29.903323  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:29.904618 10261 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.905145  9859 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:29.909066 10124 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:29.912806  7828 catalog_manager.cc:5484] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 11 to 12, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:29.918357  7826 catalog_manager.cc:5484] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 6 to 7, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.919270  7969 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:29.920672 12604 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:29.922423  8081 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac with delete_type TABLET_DATA_TOMBSTONED (TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54670
I20240503 02:13:29.923135 12575 tablet_replica.cc:330] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: stopping tablet replica
I20240503 02:13:29.923377 12575 raft_consensus.cc:2229] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:29.923815 12575 raft_consensus.cc:2258] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:29.923657  7967 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:29.924935 12575 ts_tablet_manager.cc:1875] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:29.930733  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80b4e6aa20f64473be8bcbf47baf66e4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:29.933072 12575 ts_tablet_manager.cc:1888] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:13:29.933248 12575 log.cc:1196] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/2d69c71d9dfd47a1a2429e7caf5e33ac
I20240503 02:13:29.934166  7816 catalog_manager.cc:4830] TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): tablet 2d69c71d9dfd47a1a2429e7caf5e33ac (table rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]) successfully deleted
I20240503 02:13:29.943467  9994 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:29.954138  8231 consensus_queue.cc:237] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:29.930117  9860 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
W20240503 02:13:29.962929  8168 consensus_peers.cc:485] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6043da7d2e584af885ff6a5bf8f06912. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.964608  8362 raft_consensus.cc:1261] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:29.965363 12604 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:29.971801 10124 raft_consensus.cc:1261] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:29.972471 12628 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:29.983734  7967 raft_consensus.cc:1261] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:29.984483 12604 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:30.004734 12649 ts_tablet_manager.cc:902] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:30.010718 12604 raft_consensus.cc:1013] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: attempt to promote peer d826d4dae2aa4e05b2f64c9b17a1126f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240503 02:13:30.012265 12604 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:30.015507  7969 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:30.019436  8362 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:30.020614 12067 tablet_copy_client.cc:308] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:30.021298  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 007b59de142c4b95966494ba097b1a09 from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:48749)
I20240503 02:13:30.021497  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 007b59de142c4b95966494ba097b1a09 from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:48749: session id = 03f99420df704238ab300ad6e58d878c-007b59de142c4b95966494ba097b1a09
I20240503 02:13:30.025049  8122 tablet_copy_source_session.cc:205] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.026561 12067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007b59de142c4b95966494ba097b1a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.034674 12649 tablet_copy_client.cc:308] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:30.035686  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet d3d4b912095244de816d3089a31b8649 from peer b85e85e631ab4e04a86bfe5a9f0aa512 ({username='slave'} at 127.0.145.6:49885)
I20240503 02:13:30.035949  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet d3d4b912095244de816d3089a31b8649 from peer b85e85e631ab4e04a86bfe5a9f0aa512 at {username='slave'} at 127.0.145.6:49885: session id = b85e85e631ab4e04a86bfe5a9f0aa512-d3d4b912095244de816d3089a31b8649
I20240503 02:13:30.038187  7828 catalog_manager.cc:5484] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 4 to 5, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:30.040585  8382 tablet_copy_source_session.cc:205] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.040163 10124 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:30.046480 11888 raft_consensus.cc:1050] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f to VOTER
I20240503 02:13:30.047642 11888 consensus_queue.cc:237] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
W20240503 02:13:30.057664  8168 consensus_peers.cc:485] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6043da7d2e584af885ff6a5bf8f06912. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.058264  7969 raft_consensus.cc:1261] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:30.059630 12649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3d4b912095244de816d3089a31b8649. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.059054 12604 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 02:13:30.060835  8362 raft_consensus.cc:1261] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:30.062479 10124 raft_consensus.cc:1261] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:30.066529 12139 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:30.067123 12139 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:30.068873 12067 tablet_copy_client.cc:783] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.069170 12067 tablet_copy_client.cc:647] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.073499 12139 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:30.073781 10124 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:30.079705  8362 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:30.082064  7828 catalog_manager.cc:5484] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:30.086198  7969 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:30.087819  8231 consensus_queue.cc:237] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
W20240503 02:13:30.091615  8168 consensus_peers.cc:485] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6043da7d2e584af885ff6a5bf8f06912. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.096299  7967 raft_consensus.cc:1261] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:30.100796 12139 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240503 02:13:30.103823 12067 tablet_copy_client.cc:523] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.105094 12139 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:30.108332 12067 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
I20240503 02:13:30.109146 10127 raft_consensus.cc:1261] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:30.109732  7968 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:30.124161  7821 catalog_manager.cc:5484] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 6 to 7, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:30.127956 12649 tablet_copy_client.cc:783] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.128592 12649 tablet_copy_client.cc:647] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.129742  8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 6043da7d2e584af885ff6a5bf8f06912 with delete_type TABLET_DATA_TOMBSTONED (TS 8d7c037d50c241b484ee843b06313a64 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37080
I20240503 02:13:30.130748 12675 tablet_replica.cc:330] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64: stopping tablet replica
I20240503 02:13:30.131090 12675 raft_consensus.cc:2229] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:30.134969 12675 raft_consensus.cc:2258] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:30.121817 12604 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:30.135267 12649 tablet_copy_client.cc:523] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.138114 10127 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:13:30.149736 12675 ts_tablet_manager.cc:1875] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:30.159575 10127 raft_consensus.cc:1203] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:13:30.170950  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6043da7d2e584af885ff6a5bf8f06912 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:30.185242  8231 consensus_queue.cc:237] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:30.188308 12114 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.188993 12114 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:30.189306 12114 ts_tablet_manager.cc:1367] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.583s	user 0.048s	sys 0.013s
I20240503 02:13:30.190320 12114 raft_consensus.cc:348] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:30.190784  9995 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:30.191725 12592 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:30.193270 12649 tablet_bootstrap.cc:492] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap starting.
I20240503 02:13:30.193926  8362 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:30.194721 12592 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:30.194978 12525 tablet_bootstrap.cc:492] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.195492 12525 tablet_bootstrap.cc:492] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap complete.
I20240503 02:13:30.197012 12525 ts_tablet_manager.cc:1367] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent bootstrapping tablet: real 0.828s	user 0.053s	sys 0.009s
I20240503 02:13:30.198482 12525 raft_consensus.cc:348] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
W20240503 02:13:30.199375  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32e527fa9b2f441cb5668435681b4ff6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.201153 12525 raft_consensus.cc:729] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Initialized, Role: LEARNER
I20240503 02:13:30.201699 12525 consensus_queue.cc:260] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:30.204321 12525 ts_tablet_manager.cc:1398] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240503 02:13:30.206804  7991 tablet_copy_service.cc:342] P d986fcfb926e47baa7d2d5b99f697334: Request end of tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-5ea56405515a40f785726bd58ec3d26a received from {username='slave'} at 127.0.145.8:52797
I20240503 02:13:30.206956  7991 tablet_copy_service.cc:433] P d986fcfb926e47baa7d2d5b99f697334: ending tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-5ea56405515a40f785726bd58ec3d26a on tablet 5ea56405515a40f785726bd58ec3d26a with peer ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:30.230119 12675 ts_tablet_manager.cc:1888] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:13:30.233831 12675 log.cc:1196] T 6043da7d2e584af885ff6a5bf8f06912 P 8d7c037d50c241b484ee843b06313a64: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/wals/6043da7d2e584af885ff6a5bf8f06912
I20240503 02:13:30.238226  7817 catalog_manager.cc:4830] TS 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): tablet 6043da7d2e584af885ff6a5bf8f06912 (table rebalance_test_table_3 [id=9af1e1dd28c144d9b52b73387f91d232]) successfully deleted
I20240503 02:13:30.241009 12114 raft_consensus.cc:729] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:30.242146 12604 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:30.241425 12114 consensus_queue.cc:260] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:30.247840  8362 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:30.256397 12114 ts_tablet_manager.cc:1398] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.067s	user 0.001s	sys 0.004s
I20240503 02:13:30.257836  9995 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:30.258558  7826 catalog_manager.cc:5484] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:30.271317  7967 consensus_queue.cc:237] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } }
I20240503 02:13:30.298547  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-4af5884849ec4ac3891ebf0f08ee2cff received from {username='slave'} at 127.0.145.5:48749
I20240503 02:13:30.298790  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session 03f99420df704238ab300ad6e58d878c-4af5884849ec4ac3891ebf0f08ee2cff on tablet 4af5884849ec4ac3891ebf0f08ee2cff with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:30.331193 10260 raft_consensus.cc:1261] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:30.331890  8100 raft_consensus.cc:1261] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:30.332754 12408 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:30.336856 12408 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:30.352247 12408 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:13:30.355517  8100 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:13:30.367198  7821 catalog_manager.cc:5484] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 6 to 7, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) added. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:30.388867  7907 consensus_peers.cc:485] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 -> Peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): Couldn't send request to peer 8d7c037d50c241b484ee843b06313a64. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.398156 10261 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:13:30.416090 12067 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.420756 10260 raft_consensus.cc:1203] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:13:30.416647 12067 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:30.424407 12067 ts_tablet_manager.cc:1367] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.316s	user 0.080s	sys 0.048s
I20240503 02:13:30.425503 12067 raft_consensus.cc:348] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:30.425870 12067 raft_consensus.cc:729] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:30.428303 12067 consensus_queue.cc:260] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:30.428992 12067 ts_tablet_manager.cc:1398] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240503 02:13:30.441625 12649 tablet_bootstrap.cc:492] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.442122 12649 tablet_bootstrap.cc:492] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap complete.
I20240503 02:13:30.442502 12649 ts_tablet_manager.cc:1367] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent bootstrapping tablet: real 0.249s	user 0.057s	sys 0.010s
I20240503 02:13:30.443531 12649 raft_consensus.cc:348] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:30.443838 12649 raft_consensus.cc:729] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Initialized, Role: LEARNER
I20240503 02:13:30.444067 12649 consensus_queue.cc:260] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:30.444690 12649 ts_tablet_manager.cc:1398] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:13:30.445467  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-d3d4b912095244de816d3089a31b8649 received from {username='slave'} at 127.0.145.6:49885
I20240503 02:13:30.445617  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-d3d4b912095244de816d3089a31b8649 on tablet d3d4b912095244de816d3089a31b8649 with peer b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:30.454669  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-007b59de142c4b95966494ba097b1a09 received from {username='slave'} at 127.0.145.5:48749
I20240503 02:13:30.454809  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session 03f99420df704238ab300ad6e58d878c-007b59de142c4b95966494ba097b1a09 on tablet 007b59de142c4b95966494ba097b1a09 with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:30.457140 12067 ts_tablet_manager.cc:902] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:30.494581  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:30.523579  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:30.524217  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:30.529616  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:30.534709 12067 tablet_copy_client.cc:308] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:30.537071  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:37535)
I20240503 02:13:30.537257  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet 2d69c71d9dfd47a1a2429e7caf5e33ac from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:37535: session id = 03f99420df704238ab300ad6e58d878c-2d69c71d9dfd47a1a2429e7caf5e33ac
I20240503 02:13:30.541205  8252 tablet_copy_source_session.cc:205] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.548396 12067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.586793 12067 tablet_copy_client.cc:783] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.587103 12067 tablet_copy_client.cc:647] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.596323 12067 tablet_copy_client.cc:523] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.600163 12067 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
I20240503 02:13:30.691355 12114 ts_tablet_manager.cc:902] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:30.695390 12649 ts_tablet_manager.cc:902] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:30.705916  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:30.725972 12506 raft_consensus.cc:1050] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f to VOTER
I20240503 02:13:30.727010 12506 consensus_queue.cc:237] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:30.746940  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:30.762773 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:30.787045 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:30.787081  9859 raft_consensus.cc:1203] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 02:13:30.788486 10124 raft_consensus.cc:1261] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:30.789247 12504 consensus_queue.cc:1035] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:30.790273  7967 raft_consensus.cc:1261] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:30.790971 12504 consensus_queue.cc:1035] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:30.798234  9860 raft_consensus.cc:1203] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:13:30.820655 12408 raft_consensus.cc:1050] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334: attempting to promote NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 to VOTER
I20240503 02:13:30.822072 12408 consensus_queue.cc:237] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:30.823086  9995 raft_consensus.cc:1203] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:13:30.849647 12771 ts_tablet_manager.cc:902] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: Initiating tablet copy from peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
W20240503 02:13:30.852519  8035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.7:39415, user_credentials={real_user=slave}} blocked reactor thread for 54252us
W20240503 02:13:30.862336  8038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.5:44891, user_credentials={real_user=slave}} blocked reactor thread for 59028.5us
I20240503 02:13:30.865026  8231 raft_consensus.cc:1261] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:30.868736 12506 consensus_queue.cc:1035] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:30.870244 12638 raft_consensus.cc:2932] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:30.874677 10260 raft_consensus.cc:1261] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:30.875548 12410 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:30.884665 12114 tablet_copy_client.cc:308] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:30.884732  9995 raft_consensus.cc:1261] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:30.885489  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet 32e527fa9b2f441cb5668435681b4ff6 from peer 03f99420df704238ab300ad6e58d878c ({username='slave'} at 127.0.145.5:37535)
I20240503 02:13:30.885581 12410 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:30.885771  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet 32e527fa9b2f441cb5668435681b4ff6 from peer 03f99420df704238ab300ad6e58d878c at {username='slave'} at 127.0.145.5:37535: session id = 03f99420df704238ab300ad6e58d878c-32e527fa9b2f441cb5668435681b4ff6
I20240503 02:13:30.888438  8362 raft_consensus.cc:1261] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:30.889079  8252 tablet_copy_source_session.cc:205] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.889019 12410 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:30.891532 12114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e527fa9b2f441cb5668435681b4ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.892448 10127 raft_consensus.cc:2932] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:30.907034  7967 raft_consensus.cc:2932] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:30.913113 12410 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:30.915030  9996 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:30.915779  7821 catalog_manager.cc:5484] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:30.923435 12649 tablet_copy_client.cc:308] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:30.924418  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet 6043da7d2e584af885ff6a5bf8f06912 from peer b85e85e631ab4e04a86bfe5a9f0aa512 ({username='slave'} at 127.0.145.6:56443)
I20240503 02:13:30.924633  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet 6043da7d2e584af885ff6a5bf8f06912 from peer b85e85e631ab4e04a86bfe5a9f0aa512 at {username='slave'} at 127.0.145.6:56443: session id = b85e85e631ab4e04a86bfe5a9f0aa512-6043da7d2e584af885ff6a5bf8f06912
I20240503 02:13:30.926474  8231 raft_consensus.cc:2932] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:30.928329  8252 tablet_copy_source_session.cc:205] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.929029  8361 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:30.933125 10260 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:30.935371  7830 catalog_manager.cc:5484] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 7 to 8, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:30.942389 12649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6043da7d2e584af885ff6a5bf8f06912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.963703 12771 tablet_copy_client.cc:308] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:37681
I20240503 02:13:30.964780  7991 tablet_copy_service.cc:140] P d986fcfb926e47baa7d2d5b99f697334: Received BeginTabletCopySession request for tablet c45c5a14b85447c78fe2a27be4392c88 from peer 8d7c037d50c241b484ee843b06313a64 ({username='slave'} at 127.0.145.4:51979)
I20240503 02:13:30.964951  7991 tablet_copy_service.cc:161] P d986fcfb926e47baa7d2d5b99f697334: Beginning new tablet copy session on tablet c45c5a14b85447c78fe2a27be4392c88 from peer 8d7c037d50c241b484ee843b06313a64 at {username='slave'} at 127.0.145.4:51979: session id = 8d7c037d50c241b484ee843b06313a64-c45c5a14b85447c78fe2a27be4392c88
I20240503 02:13:30.968451  7991 tablet_copy_source_session.cc:205] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.972031 12771 ts_tablet_manager.cc:1875] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:13:30.977227 12114 tablet_copy_client.cc:783] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.977579 12114 tablet_copy_client.cc:647] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.984673  8100 consensus_queue.cc:237] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:30.986780 12114 tablet_copy_client.cc:523] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.990862  7967 raft_consensus.cc:1261] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:30.990862 10127 raft_consensus.cc:1261] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:30.991793 12638 consensus_queue.cc:1035] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:30.992419 12638 consensus_queue.cc:1035] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:30.995839 12773 raft_consensus.cc:2932] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:30.998718 12114 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c: Bootstrap starting.
I20240503 02:13:31.009644  7968 raft_consensus.cc:2932] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:31.015519  7967 consensus_queue.cc:237] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:31.018965  7827 catalog_manager.cc:5484] T 8a8affdde4d8417f95f0585dd01e7d59 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 5 to 6, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:31.019256  9995 raft_consensus.cc:1261] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:31.020707 12410 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:31.024055  8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a8affdde4d8417f95f0585dd01e7d59 with delete_type TABLET_DATA_TOMBSTONED (TS 2942efc0ecb448d2ba2f6bca20fbb29c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35426
I20240503 02:13:31.024489  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a8affdde4d8417f95f0585dd01e7d59 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:31.025486 12649 tablet_copy_client.cc:783] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:31.025822 12649 tablet_copy_client.cc:647] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:31.032439 12649 tablet_copy_client.cc:523] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:31.043421 12808 tablet_replica.cc:330] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c: stopping tablet replica
I20240503 02:13:31.043853 12808 raft_consensus.cc:2229] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:31.044289 12808 raft_consensus.cc:2258] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:31.046092 12410 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:31.046911 12808 ts_tablet_manager.cc:1875] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:31.047700 10261 raft_consensus.cc:1261] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:31.050418 12408 consensus_queue.cc:1035] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:31.052569  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5ea56405515a40f785726bd58ec3d26a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:31.055264 10260 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:31.055392  9995 raft_consensus.cc:2932] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:31.057904 12649 tablet_bootstrap.cc:492] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap starting.
I20240503 02:13:31.063033 12808 ts_tablet_manager.cc:1888] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:31.062831  7828 catalog_manager.cc:5484] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 8 to 9, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:31.063283 12808 log.cc:1196] T 8a8affdde4d8417f95f0585dd01e7d59 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/8a8affdde4d8417f95f0585dd01e7d59
I20240503 02:13:31.064579 12771 ts_tablet_manager.cc:1888] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:13:31.066416 12771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45c5a14b85447c78fe2a27be4392c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:31.067721  7814 catalog_manager.cc:4830] TS 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): tablet 8a8affdde4d8417f95f0585dd01e7d59 (table rebalance_test_table_1 [id=d4270f8900eb45459ee1e49c30ff6cff]) successfully deleted
I20240503 02:13:31.070890 10127 raft_consensus.cc:2932] T 8a8affdde4d8417f95f0585dd01e7d59 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:31.086133  8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea56405515a40f785726bd58ec3d26a with delete_type TABLET_DATA_TOMBSTONED (TS 8d7c037d50c241b484ee843b06313a64 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37080
I20240503 02:13:31.099107 12771 tablet_copy_client.cc:783] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:31.099534 12771 tablet_copy_client.cc:647] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:31.102813 12771 tablet_copy_client.cc:523] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:31.116703 12067 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:31.117331 12067 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:31.120692 12067 ts_tablet_manager.cc:1367] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.521s	user 0.063s	sys 0.010s
I20240503 02:13:31.121758 12067 raft_consensus.cc:348] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:31.122141 12067 raft_consensus.cc:729] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:31.122399 12067 consensus_queue.cc:260] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:31.124274 12067 ts_tablet_manager.cc:1398] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:13:31.125102  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-2d69c71d9dfd47a1a2429e7caf5e33ac received from {username='slave'} at 127.0.145.5:37535
I20240503 02:13:31.125298  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session 03f99420df704238ab300ad6e58d878c-2d69c71d9dfd47a1a2429e7caf5e33ac on tablet 2d69c71d9dfd47a1a2429e7caf5e33ac with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:31.145471 12812 tablet_replica.cc:330] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64: stopping tablet replica
I20240503 02:13:31.145831 12812 raft_consensus.cc:2229] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:31.146287 12812 raft_consensus.cc:2258] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:31.147476 12812 ts_tablet_manager.cc:1875] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:31.155563 12812 ts_tablet_manager.cc:1888] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:31.156828 12812 log.cc:1196] T 5ea56405515a40f785726bd58ec3d26a P 8d7c037d50c241b484ee843b06313a64: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/wals/5ea56405515a40f785726bd58ec3d26a
I20240503 02:13:31.158690 12771 tablet_bootstrap.cc:492] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:13:31.166993  7817 catalog_manager.cc:4830] TS 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): tablet 5ea56405515a40f785726bd58ec3d26a (table rebalance_test_table_4 [id=c7d72eaa895849fcb02235c1b6e8d08f]) successfully deleted
I20240503 02:13:31.195890 12638 raft_consensus.cc:1050] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER 03f99420df704238ab300ad6e58d878c to VOTER
I20240503 02:13:31.196882 12638 consensus_queue.cc:237] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:31.202623 12260 raft_consensus.cc:1050] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 to VOTER
I20240503 02:13:31.203931 12260 consensus_queue.cc:237] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:31.213240 12506 raft_consensus.cc:1050] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER 03f99420df704238ab300ad6e58d878c to VOTER
I20240503 02:13:31.214221 12506 consensus_queue.cc:237] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:31.235805  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:31.243134  9995 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:31.243505  8231 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:31.243717  9860 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:31.243885 12773 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:31.244482 12774 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:31.244642 10260 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:31.244809 12506 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:31.246137  7967 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:31.247273 12774 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:31.247933 12774 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:31.249753  9860 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:31.250420 12773 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:31.255515  8230 raft_consensus.cc:1261] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:31.256502 10127 raft_consensus.cc:1261] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:31.257665 12264 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:31.258534 12264 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:31.263175  9996 raft_consensus.cc:1261] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:31.264974 12264 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:31.301419 12772 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:31.307473 12504 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:31.308372  7827 catalog_manager.cc:5484] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:31.315394 10260 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:31.316305  7827 catalog_manager.cc:5484] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:31.317787  9994 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:31.314087  7967 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:31.321731  8231 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:31.361037 12260 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:31.365897  8230 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:31.368991  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:31.369205  9996 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:31.372153  7828 catalog_manager.cc:5484] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 7 to 8, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:31.377413 10127 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:31.381867  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:31.386103  8361 consensus_queue.cc:237] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:31.375427 12649 tablet_bootstrap.cc:492] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:31.388330 12649 tablet_bootstrap.cc:492] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap complete.
I20240503 02:13:31.388641 12649 ts_tablet_manager.cc:1367] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent bootstrapping tablet: real 0.331s	user 0.051s	sys 0.019s
I20240503 02:13:31.389626 12649 raft_consensus.cc:348] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:31.389943 12649 raft_consensus.cc:729] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Initialized, Role: LEARNER
I20240503 02:13:31.390218 12649 consensus_queue.cc:260] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:13:31.398494 10124 raft_consensus.cc:1261] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:31.400892 12649 ts_tablet_manager.cc:1398] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.000s
I20240503 02:13:31.401648  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-6043da7d2e584af885ff6a5bf8f06912 received from {username='slave'} at 127.0.145.6:56443
I20240503 02:13:31.401820  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-6043da7d2e584af885ff6a5bf8f06912 on tablet 6043da7d2e584af885ff6a5bf8f06912 with peer b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:31.400384 12633 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240503 02:13:31.406505  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:31.409932 12260 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:31.412286 10124 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:31.421854  7828 catalog_manager.cc:5484] T d3d4b912095244de816d3089a31b8649 P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: config changed from index 8 to 9, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:31.424552 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:31.432132  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3d4b912095244de816d3089a31b8649 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:31.435489  8211 tablet_service.cc:1514] Processing DeleteTablet for tablet d3d4b912095244de816d3089a31b8649 with delete_type TABLET_DATA_TOMBSTONED (TS 2942efc0ecb448d2ba2f6bca20fbb29c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35426
I20240503 02:13:31.452148  9995 raft_consensus.cc:1261] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:31.453266 12260 consensus_queue.cc:1035] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:31.455520  9995 raft_consensus.cc:2932] T d3d4b912095244de816d3089a31b8649 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:31.468997 12808 tablet_replica.cc:330] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c: stopping tablet replica
I20240503 02:13:31.469283 12808 raft_consensus.cc:2229] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:31.469820 12808 raft_consensus.cc:2258] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:31.471284 12808 ts_tablet_manager.cc:1875] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:31.479475 12808 ts_tablet_manager.cc:1888] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:31.479828 12808 log.cc:1196] T d3d4b912095244de816d3089a31b8649 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/d3d4b912095244de816d3089a31b8649
I20240503 02:13:31.479955  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:31.481367  9859 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:31.494496  7814 catalog_manager.cc:4830] TS 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): tablet d3d4b912095244de816d3089a31b8649 (table rebalance_test_table_4 [id=c7d72eaa895849fcb02235c1b6e8d08f]) successfully deleted
I20240503 02:13:31.500355  9860 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:31.507223  8100 consensus_queue.cc:237] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:31.512503  9860 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:31.513109 12773 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:31.515220 10260 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:31.515851 12774 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:31.515866 12773 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:31.517414 10261 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:31.522208  9860 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:31.523406  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 007b59de142c4b95966494ba097b1a09 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:31.532518 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:31.524647  7828 catalog_manager.cc:5484] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 8 to 9, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:31.544778  7950 tablet_service.cc:1514] Processing DeleteTablet for tablet 007b59de142c4b95966494ba097b1a09 with delete_type TABLET_DATA_TOMBSTONED (TS d986fcfb926e47baa7d2d5b99f697334 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34494
I20240503 02:13:31.562525 12852 tablet_replica.cc:330] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334: stopping tablet replica
I20240503 02:13:31.562794 12852 raft_consensus.cc:2229] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:31.563160 12852 raft_consensus.cc:2258] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:31.569015 12852 ts_tablet_manager.cc:1875] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:31.598703  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:31.615914 12852 ts_tablet_manager.cc:1888] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:31.626147 12852 log.cc:1196] T 007b59de142c4b95966494ba097b1a09 P d986fcfb926e47baa7d2d5b99f697334: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/007b59de142c4b95966494ba097b1a09
I20240503 02:13:31.628944  9860 raft_consensus.cc:1203] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:13:31.658460 12114 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:31.685029 12114 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c: Bootstrap complete.
I20240503 02:13:31.685384 12114 ts_tablet_manager.cc:1367] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c: Time spent bootstrapping tablet: real 0.687s	user 0.054s	sys 0.014s
I20240503 02:13:31.686214 12114 raft_consensus.cc:348] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:31.686502 12114 raft_consensus.cc:729] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Initialized, Role: LEARNER
I20240503 02:13:31.686723 12114 consensus_queue.cc:260] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:31.692075 12114 ts_tablet_manager.cc:1398] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240503 02:13:31.697265  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session 03f99420df704238ab300ad6e58d878c-32e527fa9b2f441cb5668435681b4ff6 received from {username='slave'} at 127.0.145.5:37535
I20240503 02:13:31.697480  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session 03f99420df704238ab300ad6e58d878c-32e527fa9b2f441cb5668435681b4ff6 on tablet 32e527fa9b2f441cb5668435681b4ff6 with peer 03f99420df704238ab300ad6e58d878c
I20240503 02:13:31.699596  7814 catalog_manager.cc:4830] TS d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): tablet 007b59de142c4b95966494ba097b1a09 (table rebalance_test_table_3 [id=9af1e1dd28c144d9b52b73387f91d232]) successfully deleted
I20240503 02:13:31.710440  9996 raft_consensus.cc:1203] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:13:31.722481 12771 tablet_bootstrap.cc:492] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:31.737135 12771 tablet_bootstrap.cc:492] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: Bootstrap complete.
I20240503 02:13:31.737496 12771 ts_tablet_manager.cc:1367] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.579s	user 0.059s	sys 0.008s
I20240503 02:13:31.751492 12771 raft_consensus.cc:348] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } }
I20240503 02:13:31.758121  9860 raft_consensus.cc:1203] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:13:31.758221 12771 raft_consensus.cc:729] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: LEARNER
I20240503 02:13:31.758476 12771 consensus_queue.cc:260] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } }
I20240503 02:13:31.759651 12771 ts_tablet_manager.cc:1398] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.022s	user 0.003s	sys 0.000s
I20240503 02:13:31.760591  7991 tablet_copy_service.cc:342] P d986fcfb926e47baa7d2d5b99f697334: Request end of tablet copy session 8d7c037d50c241b484ee843b06313a64-c45c5a14b85447c78fe2a27be4392c88 received from {username='slave'} at 127.0.145.4:51979
I20240503 02:13:31.761334  7991 tablet_copy_service.cc:433] P d986fcfb926e47baa7d2d5b99f697334: ending tablet copy session 8d7c037d50c241b484ee843b06313a64-c45c5a14b85447c78fe2a27be4392c88 on tablet c45c5a14b85447c78fe2a27be4392c88 with peer 8d7c037d50c241b484ee843b06313a64
I20240503 02:13:31.796535  7968 consensus_queue.cc:237] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
W20240503 02:13:31.813771  7906 consensus_peers.cc:485] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cab70b62bdc747a9b653bffde680f37e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:31.815332  9996 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:31.816115 12408 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:31.829147  8100 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:31.835501 12408 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20240503 02:13:31.836827 12411 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:31.839850  9996 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:31.855446  7828 catalog_manager.cc:5484] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 6 to 7, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:31.915324  8101 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:13:32.049602  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:32.111153  8231 consensus_queue.cc:237] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } }
I20240503 02:13:32.134797  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:32.136777  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:32.166446  8362 raft_consensus.cc:1203] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:13:32.205999  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:13:32.209553  8167 consensus_peers.cc:485] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:32.210788  8081 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:32.219683 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:32.220770 12863 raft_consensus.cc:1050] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 to VOTER
I20240503 02:13:32.221648 12863 consensus_queue.cc:237] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:32.236470 10260 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:32.239739 12861 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:32.246927 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:32.252076  9860 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:32.252740 12861 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:32.257719 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:32.267163  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:32.276543 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:32.294598  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:32.304631 12864 raft_consensus.cc:1050] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER 03f99420df704238ab300ad6e58d878c to VOTER
I20240503 02:13:32.305577 12864 consensus_queue.cc:237] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:32.309607  7968 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:32.311980 12861 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:32.321768  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:32.323194  9996 raft_consensus.cc:1261] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:32.326031 10127 raft_consensus.cc:1261] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:32.328552 12861 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:32.330766 12861 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:32.339816  7967 raft_consensus.cc:1261] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:32.341682 12863 consensus_queue.cc:1035] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:32.371703  9859 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:32.372431 12863 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:32.377143  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:32.389654 12971 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:32.393246  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:32.398172 12863 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:32.401309  9860 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:32.415904  9995 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:32.418954  9996 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
W20240503 02:13:32.420269  8035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.7:39415, user_credentials={real_user=slave}} blocked reactor thread for 134327us
W20240503 02:13:32.425853  8038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.4:35377, user_credentials={real_user=slave}} blocked reactor thread for 145039us
I20240503 02:13:32.430315 12864 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:32.430784 12975 ts_tablet_manager.cc:902] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681)
I20240503 02:13:32.443544  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:32.413554 10260 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:32.451347  8362 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:32.460832 12979 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:32.462318  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:32.462375 12971 raft_consensus.cc:1050] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER 03f99420df704238ab300ad6e58d878c to VOTER
I20240503 02:13:32.463619 12971 consensus_queue.cc:237] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } }
W20240503 02:13:32.465421  8165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.7:39415, user_credentials={real_user=slave}} blocked reactor thread for 73087us
I20240503 02:13:32.467090  7827 catalog_manager.cc:5484] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 7 to 8, NON_VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:32.468091  7827 catalog_manager.cc:5484] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 7 to 8, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:32.486063 12975 tablet_copy_client.cc:308] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:37681
I20240503 02:13:32.486927  7991 tablet_copy_service.cc:140] P d986fcfb926e47baa7d2d5b99f697334: Received BeginTabletCopySession request for tablet cab70b62bdc747a9b653bffde680f37e from peer ca05c9a8bee148dab8a2a982b8fa2ea4 ({username='slave'} at 127.0.145.8:52797)
I20240503 02:13:32.487169  7991 tablet_copy_service.cc:161] P d986fcfb926e47baa7d2d5b99f697334: Beginning new tablet copy session on tablet cab70b62bdc747a9b653bffde680f37e from peer ca05c9a8bee148dab8a2a982b8fa2ea4 at {username='slave'} at 127.0.145.8:52797: session id = ca05c9a8bee148dab8a2a982b8fa2ea4-cab70b62bdc747a9b653bffde680f37e
W20240503 02:13:32.488878  8167 consensus_peers.cc:485] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:32.490521  7991 tablet_copy_source_session.cc:205] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:32.491322  7969 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:32.491931 12977 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:32.493165  7969 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:32.493288 10127 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:32.501152  7967 raft_consensus.cc:2932] T 6043da7d2e584af885ff6a5bf8f06912 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:13:32.504829 10258 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:32.504508 12975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cab70b62bdc747a9b653bffde680f37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:32.505697 12864 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:32.507725 12863 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:32.513731  8362 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:32.519006 12971 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:32.525218  7969 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:32.525198 10258 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:32.532339  9859 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:32.544359  9994 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:32.554852  7826 catalog_manager.cc:5484] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 7 to 8, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:32.576319  9861 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:32.577289 12978 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:32.579711 12975 tablet_copy_client.cc:783] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:32.580034 12975 tablet_copy_client.cc:647] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:32.581620  9860 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:32.588064 12975 tablet_copy_client.cc:523] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:32.609683  8362 consensus_queue.cc:237] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } }
I20240503 02:13:32.625500 12975 tablet_bootstrap.cc:492] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap starting.
W20240503 02:13:32.633459  8295 consensus_peers.cc:485] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 -> Peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Couldn't send request to peer d986fcfb926e47baa7d2d5b99f697334. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:32.635293  8231 raft_consensus.cc:1261] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:32.636080 10258 raft_consensus.cc:1261] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:32.637223 12264 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:32.637409  7825 catalog_manager.cc:5484] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 8 to 9, 03f99420df704238ab300ad6e58d878c (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:32.639241 12264 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:32.645133 12411 raft_consensus.cc:1050] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334: attempting to promote NON_VOTER 8d7c037d50c241b484ee843b06313a64 to VOTER
I20240503 02:13:32.647567 12411 consensus_queue.cc:237] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } }
I20240503 02:13:32.649536  8230 consensus_queue.cc:237] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:32.653535  9864 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:32.654356 12978 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:32.657255 12978 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:32.660555 12264 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } } }
I20240503 02:13:32.662130  9864 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:32.664211  8230 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } } }
I20240503 02:13:32.665772  9994 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:32.668876 12978 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:32.670753  9994 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:32.685113  7829 catalog_manager.cc:5484] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 6 to 7, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:32.686106  7823 catalog_manager.cc:5484] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 8 to 9, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } }
I20240503 02:13:32.687490  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 32e527fa9b2f441cb5668435681b4ff6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:32.695896  8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 32e527fa9b2f441cb5668435681b4ff6 with delete_type TABLET_DATA_TOMBSTONED (TS 8d7c037d50c241b484ee843b06313a64 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37080
I20240503 02:13:32.731904 10258 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } } }
I20240503 02:13:32.733436 13017 tablet_replica.cc:330] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64: stopping tablet replica
I20240503 02:13:32.737449 13017 raft_consensus.cc:2229] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:32.743228  8101 raft_consensus.cc:1261] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:32.743868 13003 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:32.749657 10258 raft_consensus.cc:1261] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:32.751969 13003 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:32.752493 13017 raft_consensus.cc:2258] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:32.754999 13017 ts_tablet_manager.cc:1875] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:32.763646 13017 ts_tablet_manager.cc:1888] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:32.763916 13017 log.cc:1196] T 32e527fa9b2f441cb5668435681b4ff6 P 8d7c037d50c241b484ee843b06313a64: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/wals/32e527fa9b2f441cb5668435681b4ff6
W20240503 02:13:32.769891  8295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.7:39415, user_credentials={real_user=slave}} blocked reactor thread for 71509.2us
I20240503 02:13:32.772202  7817 catalog_manager.cc:4830] TS 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): tablet 32e527fa9b2f441cb5668435681b4ff6 (table rebalance_test_table_1 [id=d4270f8900eb45459ee1e49c30ff6cff]) successfully deleted
I20240503 02:13:32.772481  8362 raft_consensus.cc:1261] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEARNER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:32.773171 13011 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:32.825881  8100 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } } }
I20240503 02:13:32.833082  7828 catalog_manager.cc:5484] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 7 to 8, 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } } }
I20240503 02:13:32.847184 13005 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } } }
I20240503 02:13:32.855437  8362 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } } }
I20240503 02:13:32.883450  7967 consensus_queue.cc:237] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } }
I20240503 02:13:32.890792 13038 ts_tablet_manager.cc:902] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:32.893832  8362 raft_consensus.cc:1261] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:32.903322  8341 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:32.878847 10260 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } } }
I20240503 02:13:32.921586 10258 raft_consensus.cc:1261] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:32.933907 13008 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.003s
I20240503 02:13:32.935884  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:32.938109 13005 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } } }
I20240503 02:13:32.940567 10258 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } } }
I20240503 02:13:32.948696  7828 catalog_manager.cc:5484] T c45c5a14b85447c78fe2a27be4392c88 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 8 to 9, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } } }
I20240503 02:13:32.954187  8081 tablet_service.cc:1514] Processing DeleteTablet for tablet c45c5a14b85447c78fe2a27be4392c88 with delete_type TABLET_DATA_TOMBSTONED (TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:54670
I20240503 02:13:32.963968  8080 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:32.969278  9844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:32.972899  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c45c5a14b85447c78fe2a27be4392c88 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:32.973289 12411 consensus_queue.cc:1035] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20240503 02:13:32.974622 13049 tablet_replica.cc:330] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: stopping tablet replica
I20240503 02:13:32.974984 13049 raft_consensus.cc:2229] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:32.975293 13049 raft_consensus.cc:2258] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:32.978670 13049 ts_tablet_manager.cc:1875] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:33.002238  8362 raft_consensus.cc:2932] T c45c5a14b85447c78fe2a27be4392c88 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } } }
I20240503 02:13:33.016116  9976 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:33.024356 13049 ts_tablet_manager.cc:1888] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:33.025032 13049 log.cc:1196] T c45c5a14b85447c78fe2a27be4392c88 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/c45c5a14b85447c78fe2a27be4392c88
I20240503 02:13:33.026225  7816 catalog_manager.cc:4830] TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): tablet c45c5a14b85447c78fe2a27be4392c88 (table rebalance_test_table_6 [id=938b875abcf9426ead26522c081d6b7c]) successfully deleted
I20240503 02:13:33.101977  8231 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false replace: true } }
I20240503 02:13:33.113751 13038 tablet_copy_client.cc:308] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:33.114917  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d ({username='slave'} at 127.0.145.2:57675)
I20240503 02:13:33.115140  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet 2d69c71d9dfd47a1a2429e7caf5e33ac from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d at {username='slave'} at 127.0.145.2:57675: session id = 9cbd8ddb1fc6479eb8cacbfd5b9bf86d-2d69c71d9dfd47a1a2429e7caf5e33ac
I20240503 02:13:33.118207  8252 tablet_copy_source_session.cc:205] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:33.121110  7969 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:33.121966 12979 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:33.127844 10241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:33.130975  8362 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:33.130573 10260 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:33.132575 12978 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:33.133181 12978 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:33.140410 12975 tablet_bootstrap.cc:492] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:33.140959 12975 tablet_bootstrap.cc:492] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap complete.
I20240503 02:13:33.141254 12975 ts_tablet_manager.cc:1367] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent bootstrapping tablet: real 0.518s	user 0.085s	sys 0.042s
I20240503 02:13:33.142112 12975 raft_consensus.cc:348] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:33.147181 12975 raft_consensus.cc:729] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Initialized, Role: LEARNER
I20240503 02:13:33.147434 12975 consensus_queue.cc:260] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:13:33.148492 12975 ts_tablet_manager.cc:1398] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20240503 02:13:33.154805  7991 tablet_copy_service.cc:342] P d986fcfb926e47baa7d2d5b99f697334: Request end of tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-cab70b62bdc747a9b653bffde680f37e received from {username='slave'} at 127.0.145.8:52797
I20240503 02:13:33.154961  7991 tablet_copy_service.cc:433] P d986fcfb926e47baa7d2d5b99f697334: ending tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-cab70b62bdc747a9b653bffde680f37e on tablet cab70b62bdc747a9b653bffde680f37e with peer ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:33.158939 13038 ts_tablet_manager.cc:1875] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:13:33.159698 13073 ts_tablet_manager.cc:902] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:33.167780 13038 ts_tablet_manager.cc:1888] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:13:33.168162 13038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:33.175598  8231 consensus_queue.cc:237] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } }
I20240503 02:13:33.176191 13038 tablet_copy_client.cc:783] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:33.176541 13038 tablet_copy_client.cc:647] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:33.178913 13038 tablet_copy_client.cc:523] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:33.192752 13038 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:33.196130  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:13:33.217157  8167 consensus_peers.cc:485] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:33.224886 10258 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:33.224895  7967 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:33.225766 12864 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:33.226316 12864 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:33.229256 12978 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:33.234148  7967 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:33.242421  7828 catalog_manager.cc:5484] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 9 to 10, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:33.245565 10258 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:33.258517  9844 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac with delete_type TABLET_DATA_TOMBSTONED (TS 03f99420df704238ab300ad6e58d878c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35050
I20240503 02:13:33.259289  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:33.282300 13093 tablet_replica.cc:330] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c: stopping tablet replica
I20240503 02:13:33.283167 13093 raft_consensus.cc:2229] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:33.285782 13093 raft_consensus.cc:2258] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:33.287240 13093 ts_tablet_manager.cc:1875] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:33.297897  7967 consensus_queue.cc:237] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } }
I20240503 02:13:33.306708 12978 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false replace: true } } }
I20240503 02:13:33.311265 13093 ts_tablet_manager.cc:1888] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:13:33.312204  8361 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false replace: true } } }
I20240503 02:13:33.319264  7828 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 9 to 10. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false replace: true } } }
I20240503 02:13:33.320900 13093 log.cc:1196] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 03f99420df704238ab300ad6e58d878c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/wals/2d69c71d9dfd47a1a2429e7caf5e33ac
I20240503 02:13:33.322830  7817 catalog_manager.cc:4830] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): tablet 2d69c71d9dfd47a1a2429e7caf5e33ac (table rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]) successfully deleted
I20240503 02:13:33.329869  7969 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false replace: true } } }
I20240503 02:13:33.345039  8231 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:33.373139 13038 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:33.380331 10108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:13:33.400846  7907 consensus_peers.cc:485] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 -> Peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): Couldn't send request to peer 8d7c037d50c241b484ee843b06313a64. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:33.403853  9994 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:33.404187 13038 tablet_bootstrap.cc:492] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap complete.
I20240503 02:13:33.404542 13038 ts_tablet_manager.cc:1367] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.212s	user 0.061s	sys 0.000s
I20240503 02:13:33.404491  8101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b"
dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59850
I20240503 02:13:33.404858  8101 raft_consensus.cc:595] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:33.404843 13008 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:33.405556 13038 raft_consensus.cc:348] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } }
I20240503 02:13:33.405867 13038 raft_consensus.cc:729] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: LEARNER
I20240503 02:13:33.406116 13038 consensus_queue.cc:260] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } }
I20240503 02:13:33.406769 13038 ts_tablet_manager.cc:1398] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:13:33.412211  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session 9cbd8ddb1fc6479eb8cacbfd5b9bf86d-2d69c71d9dfd47a1a2429e7caf5e33ac received from {username='slave'} at 127.0.145.2:57675
I20240503 02:13:33.412429  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session 9cbd8ddb1fc6479eb8cacbfd5b9bf86d-2d69c71d9dfd47a1a2429e7caf5e33ac on tablet 2d69c71d9dfd47a1a2429e7caf5e33ac with peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d
I20240503 02:13:33.414386 10260 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:33.415170 13011 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:33.422968 13053 raft_consensus.cc:979] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: : Instructing follower 03f99420df704238ab300ad6e58d878c to start an election
I20240503 02:13:33.423171 13053 raft_consensus.cc:1067] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Signalling peer 03f99420df704238ab300ad6e58d878c to start an election
I20240503 02:13:33.426847  9860 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b"
dest_uuid: "03f99420df704238ab300ad6e58d878c"
 from {username='slave'} at 127.0.145.2:34701
I20240503 02:13:33.427248  9860 raft_consensus.cc:482] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:33.427528  9860 raft_consensus.cc:3037] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.431228  8101 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:33.432297 13011 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:33.435863  9860 raft_consensus.cc:504] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:33.439864  8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "03f99420df704238ab300ad6e58d878c" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:33.440434  8100 raft_consensus.cc:3032] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:33.440553  8100 raft_consensus.cc:729] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:13:33.440946  8100 consensus_queue.cc:260] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:33.441511  8100 raft_consensus.cc:3037] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.441810 13011 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:13:33.445780  8100 raft_consensus.cc:2454] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03f99420df704238ab300ad6e58d878c in term 2.
I20240503 02:13:33.445920  8101 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:13:33.453212  9994 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:13:33.453496 10258 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:13:33.458001  9860 leader_election.cc:288] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [CANDIDATE]: Term 2 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
I20240503 02:13:33.458973  8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "03f99420df704238ab300ad6e58d878c" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:33.459244  8362 raft_consensus.cc:3037] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.461328  7828 catalog_manager.cc:5484] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 7 to 8, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) added. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:13:33.467123 13008 raft_consensus.cc:1050] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334: attempting to promote NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 to VOTER
I20240503 02:13:33.468051 13008 consensus_queue.cc:237] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } }
I20240503 02:13:33.469960  8361 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:33.470572 10260 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:33.470711 12978 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:33.472396  8101 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:33.472754 12978 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 02:13:33.474035  7907 consensus_peers.cc:485] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 -> Peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): Couldn't send request to peer 8d7c037d50c241b484ee843b06313a64. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:33.474031 10261 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false replace: true } } }
I20240503 02:13:33.475286 13005 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:33.478503  7967 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:33.479071 12978 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:33.482239 12978 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:33.486979 10259 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:33.489624 10260 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:33.494673  7814 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7e966caa651e4e42890be5d79f851a1f with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:13:33.497808  7969 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:33.499686 13011 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:33.507750  7830 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 10 to 11, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:33.515360  8360 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:33.529862  8231 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:33.533236 10258 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:13:33.533923 12978 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:13:33.539076  8360 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:13:33.540369 12979 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:13:33.540958 12979 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:33.544701 10258 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:33.553611  8361 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:33.559048  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e966caa651e4e42890be5d79f851a1f with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:33.562047  7823 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 11 to 12, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:33.567329  7950 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e966caa651e4e42890be5d79f851a1f with delete_type TABLET_DATA_TOMBSTONED (TS d986fcfb926e47baa7d2d5b99f697334 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:34494
I20240503 02:13:33.569281 13129 tablet_replica.cc:330] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334: stopping tablet replica
I20240503 02:13:33.570796 13129 raft_consensus.cc:2229] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:33.571756 13129 raft_consensus.cc:2258] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:33.570219 13011 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:13:33.574069 13129 ts_tablet_manager.cc:1875] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:33.583206  9995 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:33.584457 13008 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:33.586879  9995 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:13:33.600760 13129 ts_tablet_manager.cc:1888] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240503 02:13:33.601050 13129 log.cc:1196] T 7e966caa651e4e42890be5d79f851a1f P d986fcfb926e47baa7d2d5b99f697334: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/7e966caa651e4e42890be5d79f851a1f
I20240503 02:13:33.602350  7814 catalog_manager.cc:4830] TS d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): tablet 7e966caa651e4e42890be5d79f851a1f (table rebalance_test_table_5 [id=1f2b1fe68a7e4b59a12d8521e9e85cbb]) successfully deleted
W20240503 02:13:33.603909  8168 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e966caa651e4e42890be5d79f851a1f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:33.634943  8362 raft_consensus.cc:2454] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03f99420df704238ab300ad6e58d878c in term 2.
I20240503 02:13:33.635572  9800 leader_election.cc:302] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 03f99420df704238ab300ad6e58d878c, 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:13:33.638571 10258 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:13:33.645035 13073 tablet_copy_client.cc:308] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:33.648310 13137 raft_consensus.cc:2781] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:33.648617  7830 catalog_manager.cc:5484] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 8 to 9, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:13:33.649914  8101 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:13:33.658260 13137 raft_consensus.cc:686] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [term 2 LEADER]: Becoming Leader. State: Replica: 03f99420df704238ab300ad6e58d878c, State: Running, Role: LEADER
I20240503 02:13:33.659329 13137 consensus_queue.cc:237] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:33.669544  7967 consensus_queue.cc:237] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:33.672605  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet 834aaba965c8443c862e98a070e36cac from peer d986fcfb926e47baa7d2d5b99f697334 ({username='slave'} at 127.0.145.1:59159)
I20240503 02:13:33.672808  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet 834aaba965c8443c862e98a070e36cac from peer d986fcfb926e47baa7d2d5b99f697334 at {username='slave'} at 127.0.145.1:59159: session id = d986fcfb926e47baa7d2d5b99f697334-834aaba965c8443c862e98a070e36cac
I20240503 02:13:33.676003  8382 tablet_copy_source_session.cc:205] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:33.676740 10258 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:33.677649 13005 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:33.682011  8101 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:33.682706 13005 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:33.685524  9994 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:33.685552 13005 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:33.688129 12411 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:33.688977 13073 ts_tablet_manager.cc:1875] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:13:33.689188  8100 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:33.702703 13073 ts_tablet_manager.cc:1888] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:13:33.703068 13073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 834aaba965c8443c862e98a070e36cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:33.707157  7823 catalog_manager.cc:5484] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 9 to 10, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:33.709410  9994 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:33.713403  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cab70b62bdc747a9b653bffde680f37e with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:33.714300 10258 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:33.716212 10124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "03f99420df704238ab300ad6e58d878c" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f"
I20240503 02:13:33.716594 10124 raft_consensus.cc:3037] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.719714 10124 raft_consensus.cc:2454] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03f99420df704238ab300ad6e58d878c in term 2.
I20240503 02:13:33.719638  7830 catalog_manager.cc:5484] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c reported cstate change: term changed from 1 to 2, leader changed from 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) to 03f99420df704238ab300ad6e58d878c (127.0.145.5). New cstate: current_term: 2 leader_uuid: "03f99420df704238ab300ad6e58d878c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:33.728876 13073 tablet_copy_client.cc:783] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:33.730100 13073 tablet_copy_client.cc:647] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:33.730363  8341 tablet_service.cc:1514] Processing DeleteTablet for tablet cab70b62bdc747a9b653bffde680f37e with delete_type TABLET_DATA_TOMBSTONED (TS 8d7c037d50c241b484ee843b06313a64 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:37080
I20240503 02:13:33.735630 13073 tablet_copy_client.cc:523] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:33.744719 13073 tablet_bootstrap.cc:492] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:33.749882  7817 catalog_manager.cc:4830] TS 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): tablet cab70b62bdc747a9b653bffde680f37e (table rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]) successfully deleted
I20240503 02:13:33.759655  8100 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:33.809839  8100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f44b2b53de84a87a8757c0d00bfed37"
dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59858
I20240503 02:13:33.810156  8100 raft_consensus.cc:595] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:33.880630 13073 tablet_bootstrap.cc:492] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:33.881120 13073 tablet_bootstrap.cc:492] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Bootstrap complete.
I20240503 02:13:33.881599 13073 ts_tablet_manager.cc:1367] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.137s	user 0.072s	sys 0.012s
I20240503 02:13:33.882545 13073 raft_consensus.cc:348] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } }
I20240503 02:13:33.882922 13073 raft_consensus.cc:729] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: LEARNER
I20240503 02:13:33.883155 13073 consensus_queue.cc:260] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } }
I20240503 02:13:33.884159 13073 ts_tablet_manager.cc:1398] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:13:33.885202  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session d986fcfb926e47baa7d2d5b99f697334-834aaba965c8443c862e98a070e36cac received from {username='slave'} at 127.0.145.1:59159
I20240503 02:13:33.885355  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session d986fcfb926e47baa7d2d5b99f697334-834aaba965c8443c862e98a070e36cac on tablet 834aaba965c8443c862e98a070e36cac with peer d986fcfb926e47baa7d2d5b99f697334
I20240503 02:13:33.899376  8361 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298"
dest_uuid: "8d7c037d50c241b484ee843b06313a64"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59948
I20240503 02:13:33.899813  8361 raft_consensus.cc:595] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:33.905512 12774 raft_consensus.cc:979] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: : Instructing follower ca05c9a8bee148dab8a2a982b8fa2ea4 to start an election
I20240503 02:13:33.905664 12774 raft_consensus.cc:1067] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Signalling peer ca05c9a8bee148dab8a2a982b8fa2ea4 to start an election
I20240503 02:13:33.906658 10258 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0f44b2b53de84a87a8757c0d00bfed37"
dest_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4"
 from {username='slave'} at 127.0.145.2:53361
I20240503 02:13:33.906971 10258 raft_consensus.cc:482] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:33.907141 10258 raft_consensus.cc:3037] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.910293 10258 raft_consensus.cc:504] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:33.914148  8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f44b2b53de84a87a8757c0d00bfed37" candidate_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:33.914433  8100 raft_consensus.cc:3032] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:33.914542  8100 raft_consensus.cc:729] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:13:33.915092  8100 consensus_queue.cc:260] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:33.915539  8100 raft_consensus.cc:3037] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.918454  8100 raft_consensus.cc:2454] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca05c9a8bee148dab8a2a982b8fa2ea4 in term 2.
I20240503 02:13:33.919942 13018 raft_consensus.cc:979] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: : Instructing follower d826d4dae2aa4e05b2f64c9b17a1126f to start an election
I20240503 02:13:33.920065 13018 raft_consensus.cc:1067] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Signalling peer d826d4dae2aa4e05b2f64c9b17a1126f to start an election
I20240503 02:13:33.921173 10127 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298"
dest_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f"
 from {username='slave'} at 127.0.145.4:34155
I20240503 02:13:33.921531 10127 raft_consensus.cc:482] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:33.921715 10127 raft_consensus.cc:3037] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.936848 10127 raft_consensus.cc:504] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:33.939069 10258 leader_election.cc:288] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [CANDIDATE]: Term 2 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891), b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397)
I20240503 02:13:33.939579  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298" candidate_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:33.939913  8361 raft_consensus.cc:3032] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:33.940021  8361 raft_consensus.cc:729] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Running, Role: LEADER
I20240503 02:13:33.940485  8361 consensus_queue.cc:260] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:33.941103  8361 raft_consensus.cc:3037] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.946403  8361 raft_consensus.cc:2454] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d826d4dae2aa4e05b2f64c9b17a1126f in term 2.
I20240503 02:13:33.948554  8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298" candidate_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:33.948949 10127 leader_election.cc:288] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [CANDIDATE]: Term 2 election: Requested vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891)
I20240503 02:13:33.949934  8100 raft_consensus.cc:3037] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.954772  8100 raft_consensus.cc:2454] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d826d4dae2aa4e05b2f64c9b17a1126f in term 2.
I20240503 02:13:33.955703 10063 leader_election.cc:302] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d826d4dae2aa4e05b2f64c9b17a1126f; no voters: 
I20240503 02:13:33.971411 13209 raft_consensus.cc:2781] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:33.986018  9860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298" candidate_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "03f99420df704238ab300ad6e58d878c"
I20240503 02:13:33.986443  9860 raft_consensus.cc:3037] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.990049  9860 raft_consensus.cc:2454] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 03f99420df704238ab300ad6e58d878c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d826d4dae2aa4e05b2f64c9b17a1126f in term 2.
I20240503 02:13:33.995215 13209 raft_consensus.cc:686] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 LEADER]: Becoming Leader. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Running, Role: LEADER
I20240503 02:13:33.995915 13209 consensus_queue.cc:237] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:33.999199  8231 consensus_queue.cc:237] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:34.004792  9994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f44b2b53de84a87a8757c0d00bfed37" candidate_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512"
I20240503 02:13:34.005190  9994 raft_consensus.cc:3037] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:34.010596  9994 raft_consensus.cc:2454] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca05c9a8bee148dab8a2a982b8fa2ea4 in term 2.
I20240503 02:13:34.011567 10197 leader_election.cc:302] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, b85e85e631ab4e04a86bfe5a9f0aa512, ca05c9a8bee148dab8a2a982b8fa2ea4; no voters: 
I20240503 02:13:34.012753 12992 raft_consensus.cc:2781] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:34.013279 12992 raft_consensus.cc:686] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 LEADER]: Becoming Leader. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Running, Role: LEADER
I20240503 02:13:34.013870 12992 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:34.013965  7830 catalog_manager.cc:5484] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: term changed from 1 to 2, leader changed from 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) to d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7). New cstate: current_term: 2 leader_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:34.017534  8168 consensus_peers.cc:485] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecb5c78a22a74728ab40f1da8f90388c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:34.019146 10127 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:34.024396 12979 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:34.025534  7967 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:34.029650  9860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f44b2b53de84a87a8757c0d00bfed37" candidate_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "03f99420df704238ab300ad6e58d878c"
I20240503 02:13:34.029980  9860 raft_consensus.cc:3037] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:34.030177 12864 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:34.034034  9860 raft_consensus.cc:2454] T 0f44b2b53de84a87a8757c0d00bfed37 P 03f99420df704238ab300ad6e58d878c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca05c9a8bee148dab8a2a982b8fa2ea4 in term 2.
I20240503 02:13:34.042398  7830 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: term changed from 1 to 2, leader changed from 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) to ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8). New cstate: current_term: 2 leader_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:34.083743 13234 ts_tablet_manager.cc:902] T 7e966caa651e4e42890be5d79f851a1f P 03f99420df704238ab300ad6e58d878c: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
F20240503 02:13:34.085976 13234 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-007b59de142c4b95966494ba097b1a09.queue_time_us, tablet-download-pool-007b59de142c4b95966494ba097b1a09.queue_time_us) (tablet-download-pool-007b59de142c4b95966494ba097b1a09.run_cpu_time_us, tablet-download-pool-007b59de142c4b95966494ba097b1a09.run_cpu_time_us) (tablet-download-pool-007b59de142c4b95966494ba097b1a09.run_wall_time_us, tablet-download-pool-007b59de142c4b95966494ba097b1a09.run_wall_time_us) (tablet-download-pool-0f44b2b53de84a87a8757c0d00bfed37.queue_time_us, tablet-download-pool-0f44b2b53de84a87a8757c0d00bfed37.queue_time_us) (tablet-download-pool-0f44b2b53de84a87a8757c0d00bfed37.run_cpu_time_us, tablet-download-pool-0f44b2b53de84a87a8757c0d00bfed37.run_cpu_time_us) (tablet-download-pool-0f44b2b53de84a87a8757c0d00bfed37.run_wall_time_us, tablet-download-pool-0f44b2b53de84a87a8757c0d00bfed37.run_wall_time_us) (tablet-download-pool-2448e12efbd44a90b1f8bce9519e73e0.queue_time_us, tablet-download-pool-2448e12efbd44a90b1f8bce9519e73e0.queue_time_us) (tablet-download-pool-2448e12efbd44a90b1f8bce9519e73e0.run_cpu_time_us, tablet-download-pool-2448e12efbd44a90b1f8bce9519e73e0.run_cpu_time_us) (tablet-download-pool-2448e12efbd44a90b1f8bce9519e73e0.run_wall_time_us, tablet-download-pool-2448e12efbd44a90b1f8bce9519e73e0.run_wall_time_us) (tablet-download-pool-2d69c71d9dfd47a1a2429e7caf5e33ac.queue_time_us, tablet-download-pool-2d69c71d9dfd47a1a2429e7caf5e33ac.queue_time_us) (tablet-download-pool-2d69c71d9dfd47a1a2429e7caf5e33ac.run_cpu_time_us, tablet-download-pool-2d69c71d9dfd47a1a2429e7caf5e33ac.run_cpu_time_us) (tablet-download-pool-2d69c71d9dfd47a1a2429e7caf5e33ac.run_wall_time_us, tablet-download-pool-2d69c71d9dfd47a1a2429e7caf5e33ac.run_wall_time_us) (tablet-download-pool-32e527fa9b2f441cb5668435681b4ff6.queue_time_us, tablet-download-pool-32e527fa9b2f441cb5668435681b4ff6.queue_time_us) (tablet-download-pool-32e527fa9b2f441cb5668435681b4ff6.run_cpu_time_us, tablet-download-pool-32e527fa9b2f441cb5668435681b4ff6.run_cpu_time_us) (tablet-download-pool-32e527fa9b2f441cb5668435681b4ff6.run_wall_time_us, tablet-download-pool-32e527fa9b2f441cb5668435681b4ff6.run_wall_time_us) (tablet-download-pool-402de43677c44deab8793da1895d4fb2.queue_time_us, tablet-download-pool-402de43677c44deab8793da1895d4fb2.queue_time_us) (tablet-download-pool-402de43677c44deab8793da1895d4fb2.run_cpu_time_us, tablet-download-pool-402de43677c44deab8793da1895d4fb2.run_cpu_time_us) (tablet-download-pool-402de43677c44deab8793da1895d4fb2.run_wall_time_us, tablet-download-pool-402de43677c44deab8793da1895d4fb2.run_wall_time_us) (tablet-download-pool-4af5884849ec4ac3891ebf0f08ee2cff.queue_time_us, tablet-download-pool-4af5884849ec4ac3891ebf0f08ee2cff.queue_time_us) (tablet-download-pool-4af5884849ec4ac3891ebf0f08ee2cff.run_cpu_time_us, tablet-download-pool-4af5884849ec4ac3891ebf0f08ee2cff.run_cpu_time_us) (tablet-download-pool-4af5884849ec4ac3891ebf0f08ee2cff.run_wall_time_us, tablet-download-pool-4af5884849ec4ac3891ebf0f08ee2cff.run_wall_time_us) (tablet-download-pool-72e2bb75da654e8ab45bba8ac077f28c.queue_time_us, tablet-download-pool-72e2bb75da654e8ab45bba8ac077f28c.queue_time_us) (tablet-download-pool-72e2bb75da654e8ab45bba8ac077f28c.run_cpu_time_us, tablet-download-pool-72e2bb75da654e8ab45bba8ac077f28c.run_cpu_time_us) (tablet-download-pool-72e2bb75da654e8ab45bba8ac077f28c.run_wall_time_us, tablet-download-pool-72e2bb75da654e8ab45bba8ac077f28c.run_wall_time_us) (tablet-download-pool-7e4acc8b5ceb4cd5b0f3708ff4f19298.queue_time_us, tablet-download-pool-7e4acc8b5ceb4cd5b0f3708ff4f19298.queue_time_us) (tablet-download-pool-7e4acc8b5ceb4cd5b0f3708ff4f19298.run_cpu_time_us, tablet-download-pool-7e4acc8b5ceb4cd5b0f3708ff4f19298.run_cpu_time_us) (tablet-download-pool-7e4acc8b5ceb4cd5b0f3708ff4f19298.run_wall_time_us, tablet-download-pool-7e4acc8b5ceb4cd5b0f3708ff4f19298.run_wall_time_us) (tablet-download-pool-7e966caa651e4e42890be5d79f851a1f.queue_time_us, tablet-download-pool-7e966caa651e4e42890be5d79f851a1f.queue_time_us) (tablet-download-pool-80b4e6aa20f64473be8bcbf47baf66e4.queue_time_us, tablet-download-pool-80b4e6aa20f64473be8bcbf47baf66e4.queue_time_us) (tablet-download-pool-80b4e6aa20f64473be8bcbf47baf66e4.run_cpu_time_us, tablet-download-pool-80b4e6aa20f64473be8bcbf47baf66e4.run_cpu_time_us) (tablet-download-pool-80b4e6aa20f64473be8bcbf47baf66e4.run_wall_time_us, tablet-download-pool-80b4e6aa20f64473be8bcbf47baf66e4.run_wall_time_us) (tablet-download-pool-aeaee03abc644807b8278cfb78e4f02b.queue_time_us, tablet-download-pool-aeaee03abc644807b8278cfb78e4f02b.queue_time_us) (tablet-download-pool-aeaee03abc644807b8278cfb78e4f02b.run_cpu_time_us, tablet-download-pool-aeaee03abc644807b8278cfb78e4f02b.run_cpu_time_us) (tablet-download-pool-aeaee03abc644807b8278cfb78e4f02b.run_wall_time_us, tablet-download-pool-aeaee03abc644807b8278cfb78e4f02b.run_wall_time_us) (tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.queue_time_us, tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.queue_time_us) (tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.run_cpu_time_us, tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.run_cpu_time_us) (tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.run_wall_time_us, tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.run_wall_time_us) (tablet-download-pool-c23b06764ce54acb8388de94aea4cd4b.queue_time_us, tablet-download-pool-c23b06764ce54acb8388de94aea4cd4b.queue_time_us) (tablet-download-pool-c23b06764ce54acb8388de94aea4cd4b.run_cpu_time_us, tablet-download-pool-c23b06764ce54acb8388de94aea4cd4b.run_cpu_time_us) (tablet-download-pool-c23b06764ce54acb8388de94aea4cd4b.run_wall_time_us, tablet-download-pool-c23b06764ce54acb8388de94aea4cd4b.run_wall_time_us) (tablet-download-pool-ceae1c7c45a34b24badbbe23cb9e0176.queue_time_us, tablet-download-pool-ceae1c7c45a34b24badbbe23cb9e0176.queue_time_us) (tablet-download-pool-ceae1c7c45a34b24badbbe23cb9e0176.run_cpu_time_us, tablet-download-pool-ceae1c7c45a34b24badbbe23cb9e0176.run_cpu_time_us) (tablet-download-pool-ceae1c7c45a34b24badbbe23cb9e0176.run_wall_time_us, tablet-download-pool-ceae1c7c45a34b24badbbe23cb9e0176.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 02:13:34.088224  7969 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:34.089494  7968 raft_consensus.cc:1203] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:13:34.086062 12979 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
I20240503 02:13:34.108214  7823 catalog_manager.cc:5484] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
    @     0x7f2e6cf53dcd  google::LogMessage::Fail() at ??:0
    @     0x7f2e6cf57b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f2e6cf537cc  google::LogMessage::Flush() at ??:0
    @     0x7f2e6cf54f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f2e6fc94742  kudu::TraceMetrics::InternName() at ??:0
I20240503 02:13:34.141530 10124 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } } }
    @     0x7f2e6fc60f8b  kudu::ThreadPool::ThreadPool() at ??:0
    @     0x7f2e6fc5acd6  kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 02:13:34.148061 10127 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Refusing update from remote peer 03f99420df704238ab300ad6e58d878c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:34.149951 12978 raft_consensus.cc:1050] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to VOTER
I20240503 02:13:34.150961 12978 consensus_queue.cc:237] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } }
    @     0x7f2e7f0ae8de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 02:13:34.154672 10258 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:34.155953  7967 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:34.159184 13139 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
    @     0x7f2e7f0c6daf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 02:13:34.162120 13248 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:34.162575  8100 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:34.167687 12979 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:34.168404 12979 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
    @     0x7f2e7f1f1480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 02:13:34.179656 12864 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:34.185618 10258 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
    @     0x7f2e7f229a8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 02:13:34.188611  7967 consensus_queue.cc:237] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:13:34.190019  8100 consensus_queue.cc:237] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } }
I20240503 02:13:34.191040 10259 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:34.191237  8101 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:34.191674  9994 raft_consensus.cc:1261] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:34.192265 12411 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:34.195128 12411 consensus_queue.cc:1035] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:34.197592  7828 catalog_manager.cc:5484] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 10 to 11, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:34.197517 12411 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:34.203982  9994 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
W20240503 02:13:34.207757  8035 consensus_peers.cc:485] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): Couldn't send request to peer 2942efc0ecb448d2ba2f6bca20fbb29c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:34.208745  7969 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:34.208930  9995 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
    @     0x7f2e7f20c981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240503 02:13:34.209512 13053 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:34.210786 13053 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:34.213752  7968 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:34.218819 10259 raft_consensus.cc:2932] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:34.226799  7828 catalog_manager.cc:5484] T cab70b62bdc747a9b653bffde680f37e P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 10 to 11, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:13:34.230965  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cab70b62bdc747a9b653bffde680f37e with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:34.231761  8081 tablet_service.cc:1514] Processing DeleteTablet for tablet cab70b62bdc747a9b653bffde680f37e with delete_type TABLET_DATA_TOMBSTONED (TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:54670
I20240503 02:13:34.249596 13262 tablet_replica.cc:330] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: stopping tablet replica
    @     0x7f2e7f20c70e  std::_Function_handler<>::_M_invoke() at ??:0
I20240503 02:13:34.249831 13262 raft_consensus.cc:2229] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:34.250183 13262 raft_consensus.cc:2258] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:34.254211  8100 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer 03f99420df704238ab300ad6e58d878c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:34.253206 13262 ts_tablet_manager.cc:1875] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:34.254943 13137 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240503 02:13:34.272569 13262 ts_tablet_manager.cc:1888] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 02:13:34.272728 13262 log.cc:1196] T cab70b62bdc747a9b653bffde680f37e P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/cab70b62bdc747a9b653bffde680f37e
I20240503 02:13:34.276376  7816 catalog_manager.cc:4830] TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): tablet cab70b62bdc747a9b653bffde680f37e (table rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]) successfully deleted
I20240503 02:13:34.291383  9994 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } } }
I20240503 02:13:34.293747 13053 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } } }
I20240503 02:13:34.320353  7969 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } } }
I20240503 02:13:34.320998  7823 catalog_manager.cc:5484] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 6 to 7, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:34.323756  8361 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer 03f99420df704238ab300ad6e58d878c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:34.324717 13137 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P 03f99420df704238ab300ad6e58d878c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
    @     0x7f2e6fc64b75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7f2e6fc6819c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240503 02:13:34.390213  8100 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:34.391252 13209 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:34.406152  9994 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 02:13:34.406946 12992 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
    @     0x7f2e6fc68019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240503 02:13:34.438546  8231 consensus_queue.cc:237] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } }
    @     0x7f2e6fc4301e  kudu::Thread::SuperviseThread() at ??:0
I20240503 02:13:34.446928  8100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b64f66ace86e42f3ac213ba85bcab584"
dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59914
I20240503 02:13:34.447170  8100 raft_consensus.cc:595] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:34.454173  8361 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:34.455511 12979 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:34.455873  9864 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 03f99420df704238ab300ad6e58d878c [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:34.457214 12979 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:13:34.460017  8167 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
    @     0x7f2e72fc86db  start_thread at ??:0
    @     0x7f2e6be0671f  clone at ??:0
I20240503 02:13:34.479971 12979 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:34.483690  8362 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:34.490239  7827 catalog_manager.cc:5484] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 6 to 7, NON_VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:34.502815  8361 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:34.503520 13209 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:34.569770  8100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b64f66ace86e42f3ac213ba85bcab584"
dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59934
I20240503 02:13:34.570003  8100 raft_consensus.cc:595] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:34.575062  8100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "72e2bb75da654e8ab45bba8ac077f28c"
dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59926
I20240503 02:13:34.575350  8100 raft_consensus.cc:595] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:34.610738 13018 raft_consensus.cc:1050] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER d986fcfb926e47baa7d2d5b99f697334 to VOTER
I20240503 02:13:34.612005 13018 consensus_queue.cc:237] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } }
I20240503 02:13:34.620231  8100 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 02:13:34.621680 12992 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:13:34.641110  8101 consensus_queue.cc:237] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } }
I20240503 02:13:34.644649  9995 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:34.644834  8231 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:13:34.645536 12774 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:34.649247 12774 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:34.652112 12774 raft_consensus.cc:979] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: : Instructing follower b85e85e631ab4e04a86bfe5a9f0aa512 to start an election
I20240503 02:13:34.652295 12774 raft_consensus.cc:1067] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Signalling peer b85e85e631ab4e04a86bfe5a9f0aa512 to start an election
I20240503 02:13:34.653237  9994 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72e2bb75da654e8ab45bba8ac077f28c"
dest_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512"
 from {username='slave'} at 127.0.145.2:59727
I20240503 02:13:34.653669  9994 raft_consensus.cc:482] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:34.653831  9994 raft_consensus.cc:3037] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:34.654206  8211 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:34.657802  9994 raft_consensus.cc:504] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:34.661010  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e2bb75da654e8ab45bba8ac077f28c" candidate_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:34.661325  8101 raft_consensus.cc:3032] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:34.661443  8101 raft_consensus.cc:729] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:13:34.661845  8101 consensus_queue.cc:260] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:34.662242  8101 raft_consensus.cc:3037] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:34.662626  9994 leader_election.cc:288] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891), d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
I20240503 02:13:34.666232  8101 raft_consensus.cc:2454] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b85e85e631ab4e04a86bfe5a9f0aa512 in term 2.
I20240503 02:13:34.693284 12774 raft_consensus.cc:979] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: : Instructing follower ca05c9a8bee148dab8a2a982b8fa2ea4 to start an election
I20240503 02:13:34.693471 12774 raft_consensus.cc:1067] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Signalling peer ca05c9a8bee148dab8a2a982b8fa2ea4 to start an election
I20240503 02:13:34.694789 10259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b64f66ace86e42f3ac213ba85bcab584"
dest_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4"
 from {username='slave'} at 127.0.145.2:53361
I20240503 02:13:34.694835 10127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e2bb75da654e8ab45bba8ac077f28c" candidate_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f"
I20240503 02:13:34.695013 10259 raft_consensus.cc:482] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:34.695060 10127 raft_consensus.cc:3037] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:34.695148 10259 raft_consensus.cc:3037] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:34.697959 10127 raft_consensus.cc:2454] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b85e85e631ab4e04a86bfe5a9f0aa512 in term 2.
I20240503 02:13:34.698148  7969 raft_consensus.cc:1261] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:34.698459 10259 raft_consensus.cc:504] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:34.699172 13016 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:34.699569 10259 leader_election.cc:288] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [CANDIDATE]: Term 2 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891)
I20240503 02:13:34.702643  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64f66ace86e42f3ac213ba85bcab584" candidate_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:34.702872  8361 raft_consensus.cc:3037] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:34.705927  8361 raft_consensus.cc:2454] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca05c9a8bee148dab8a2a982b8fa2ea4 in term 2.
I20240503 02:13:34.707347  9929 leader_election.cc:302] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, b85e85e631ab4e04a86bfe5a9f0aa512, d826d4dae2aa4e05b2f64c9b17a1126f; no voters: 
I20240503 02:13:34.707482  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64f66ace86e42f3ac213ba85bcab584" candidate_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:34.707733  8101 raft_consensus.cc:3032] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:34.707818  8101 raft_consensus.cc:729] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:13:34.707966 13263 raft_consensus.cc:2781] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:34.708135  8101 consensus_queue.cc:260] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:34.708606  8101 raft_consensus.cc:3037] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:34.712963 10259 raft_consensus.cc:1261] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:34.713997 13016 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:34.715907  8231 raft_consensus.cc:1261] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:34.716682 13016 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:34.731741 13018 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:34.732424  8101 raft_consensus.cc:2454] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca05c9a8bee148dab8a2a982b8fa2ea4 in term 2.
I20240503 02:13:34.733000 10196 leader_election.cc:302] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, ca05c9a8bee148dab8a2a982b8fa2ea4; no voters: 
I20240503 02:13:34.733397 12992 raft_consensus.cc:2781] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:34.750339 13361 ts_tablet_manager.cc:902] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:34.755365 12992 raft_consensus.cc:686] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 LEADER]: Becoming Leader. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Running, Role: LEADER
I20240503 02:13:34.756707 12992 consensus_queue.cc:237] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:34.757279 13273 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } }
I20240503 02:13:34.763542  9995 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } }
I20240503 02:13:34.766683  7823 catalog_manager.cc:5484] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 9 to 10. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:34.777704 13263 raft_consensus.cc:686] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 LEADER]: Becoming Leader. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Running, Role: LEADER
I20240503 02:13:34.778295  7826 catalog_manager.cc:5484] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: term changed from 1 to 2, leader changed from 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) to ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8). New cstate: current_term: 2 leader_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:34.778430 13263 consensus_queue.cc:237] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:13:34.784940  8101 consensus_queue.cc:237] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } }
I20240503 02:13:34.800729  9976 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:13:34.808423 10108 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:13:34.835862  8035 consensus_peers.cc:485] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Couldn't send request to peer d986fcfb926e47baa7d2d5b99f697334. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:34.842195  9994 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:34.879222  8081 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:13:34.895326 13273 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:34.803061  7969 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:34.797397  8232 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } }
I20240503 02:13:34.841809  8231 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:34.921473  8229 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d69c71d9dfd47a1a2429e7caf5e33ac"
dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37416
I20240503 02:13:34.944025 13361 tablet_copy_client.cc:308] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:34.970528  8229 raft_consensus.cc:595] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:34.971338  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 45fe6ec58eed40f1898d4a2f5727cfa1 from peer 2942efc0ecb448d2ba2f6bca20fbb29c ({username='slave'} at 127.0.145.3:57873)
I20240503 02:13:34.971519  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 45fe6ec58eed40f1898d4a2f5727cfa1 from peer 2942efc0ecb448d2ba2f6bca20fbb29c at {username='slave'} at 127.0.145.3:57873: session id = 2942efc0ecb448d2ba2f6bca20fbb29c-45fe6ec58eed40f1898d4a2f5727cfa1
I20240503 02:13:34.976787  8230 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:34.977768 13273 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:34.979707  8122 tablet_copy_source_session.cc:205] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 02:13:34.986590  8038 connection.cc:535] client connection to 127.0.145.5:44891 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:34.988337 10064 connection.cc:535] client connection to 127.0.145.5:44891 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:35.116958  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:35.117070 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:13:34.986898 13273 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } } }
I20240503 02:13:34.988246 13412 ts_tablet_manager.cc:902] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
W20240503 02:13:35.122884 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:35.151162  8038 consensus_peers.cc:485] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:35.153836  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:35.155874 13412 tablet_copy_client.cc:308] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:13:35.181224 10127 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:13:34.848117  8341 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:13:34.992107 13357 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:13:34.660326 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.145.5:44891 (local address 127.0.145.6:56129)
0503 02:13:34.662401 (+  2075us) negotiation.cc:107] Waiting for socket to connect
0503 02:13:34.662422 (+    21us) client_negotiation.cc:174] Beginning negotiation
0503 02:13:34.679703 (+ 17281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:13:34.991036 (+311333us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1955,"thread_start_us":1793,"threads_started":1}
I20240503 02:13:34.920601  8361 consensus_queue.cc:237] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } }
W20240503 02:13:35.193429  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:13:35.141867  9995 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } } }
W20240503 02:13:34.987143  8298 connection.cc:535] client connection to 127.0.145.5:44891 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:35.194860  8298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:35.213937  8297 consensus_peers.cc:485] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:34.980459  7950 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:34.801415  7826 catalog_manager.cc:5484] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: term changed from 1 to 2, leader changed from 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) to b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6). New cstate: current_term: 2 leader_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 02:13:35.237131  8298 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:34.984120 13361 ts_tablet_manager.cc:1875] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240503 02:13:35.246730  8298 consensus_peers.cc:485] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:34.986224  8168 connection.cc:535] client connection to 127.0.145.5:44891 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:35.251555  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:13:34.847728  7829 catalog_manager.cc:5484] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 7 to 8, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:35.253538 13361 ts_tablet_manager.cc:1888] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:13:35.254063 13361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45fe6ec58eed40f1898d4a2f5727cfa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:35.182286 13263 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:13:34.839286 10259 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:34.991040  8230 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } } }
I20240503 02:13:35.121450  7816 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4af5884849ec4ac3891ebf0f08ee2cff with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 02:13:35.259675  7816 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.145.2:35309, user_credentials={real_user=slave}} blocked reactor thread for 138195us
W20240503 02:13:35.207840  9932 leader_election.cc:334] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:35.236243  9995 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240503 02:13:35.260746  9932 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.145.5:44891, user_credentials={real_user=slave}} blocked reactor thread for 52903.3us
I20240503 02:13:35.261090 13018 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 02:13:34.985319 10197 connection.cc:535] client connection to 127.0.145.5:44891 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:35.261899 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:13:35.156638  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet 2448e12efbd44a90b1f8bce9519e73e0 from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d ({username='slave'} at 127.0.145.2:57675)
I20240503 02:13:35.215796 13486 raft_consensus.cc:979] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: : Instructing follower 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to start an election
I20240503 02:13:35.262355  8101 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:13:35.262461  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet 2448e12efbd44a90b1f8bce9519e73e0 from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d at {username='slave'} at 127.0.145.2:57675: session id = 9cbd8ddb1fc6479eb8cacbfd5b9bf86d-2448e12efbd44a90b1f8bce9519e73e0
I20240503 02:13:35.262552 13486 raft_consensus.cc:1067] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Signalling peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to start an election
I20240503 02:13:35.263783  8101 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2d69c71d9dfd47a1a2429e7caf5e33ac"
dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
 from {username='slave'} at 127.0.145.3:57873
I20240503 02:13:35.264060  8101 raft_consensus.cc:482] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:35.264158  8101 raft_consensus.cc:3037] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:35.265554  8252 tablet_copy_source_session.cc:205] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:35.125437  7828 catalog_manager.cc:5484] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 10 to 11, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:35.266898 13412 ts_tablet_manager.cc:1875] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:13:35.272291  8101 raft_consensus.cc:504] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } }
I20240503 02:13:35.273633  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d69c71d9dfd47a1a2429e7caf5e33ac" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:35.273866  7969 raft_consensus.cc:3037] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:35.275822 13363 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:13:35.280068  7969 raft_consensus.cc:2454] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 2.
I20240503 02:13:35.281600  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d69c71d9dfd47a1a2429e7caf5e33ac" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:35.281844  8232 raft_consensus.cc:3032] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:35.281924  8232 raft_consensus.cc:729] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Running, Role: LEADER
I20240503 02:13:35.282269  8232 consensus_queue.cc:260] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } }
I20240503 02:13:35.282716  8232 raft_consensus.cc:3037] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:35.287781  8361 consensus_queue.cc:237] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } }
I20240503 02:13:35.291764  7968 raft_consensus.cc:1261] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:35.292757  8232 raft_consensus.cc:2454] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 2.
I20240503 02:13:35.293411  8035 leader_election.cc:302] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:35.293805 13273 raft_consensus.cc:2781] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:35.293956 13273 raft_consensus.cc:686] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 LEADER]: Becoming Leader. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:13:35.294293 13273 consensus_queue.cc:237] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } }
I20240503 02:13:35.295377 13369 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:35.300556 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d69c71d9dfd47a1a2429e7caf5e33ac" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4"
I20240503 02:13:35.300910 10259 raft_consensus.cc:3037] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:35.301481 13018 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:35.306675  7968 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
W20240503 02:13:35.310266  8168 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.5:44891, user_credentials={real_user=slave}} blocked reactor thread for 58789us
I20240503 02:13:35.311013  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 834aaba965c8443c862e98a070e36cac with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:35.313225 13361 tablet_copy_client.cc:783] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:35.315060 13361 tablet_copy_client.cc:647] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:35.315788  8101 leader_election.cc:288] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 2 election: Requested vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
I20240503 02:13:35.318653 13361 tablet_copy_client.cc:523] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:35.318956 10259 raft_consensus.cc:2454] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 2.
I20240503 02:13:35.322525 13412 ts_tablet_manager.cc:1888] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:13:35.326942  7823 catalog_manager.cc:5484] T 834aaba965c8443c862e98a070e36cac P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 8 to 9, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:35.331689  8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 834aaba965c8443c862e98a070e36cac with delete_type TABLET_DATA_TOMBSTONED (TS 2942efc0ecb448d2ba2f6bca20fbb29c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35426
W20240503 02:13:35.337811  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:35.340225  7828 catalog_manager.cc:5484] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: term changed from 1 to 2, leader changed from 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) to 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2). New cstate: current_term: 2 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:35.342187  8230 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e966caa651e4e42890be5d79f851a1f"
dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37496
I20240503 02:13:35.342430  8230 raft_consensus.cc:595] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:35.349283 13514 tablet_replica.cc:330] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c: stopping tablet replica
I20240503 02:13:35.349607 13514 raft_consensus.cc:2229] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:35.351375 13361 tablet_bootstrap.cc:492] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:13:35.355044 13514 raft_consensus.cc:2258] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:35.356730 12864 raft_consensus.cc:979] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c: : Instructing follower ca05c9a8bee148dab8a2a982b8fa2ea4 to start an election
I20240503 02:13:35.356866 12864 raft_consensus.cc:1067] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Signalling peer ca05c9a8bee148dab8a2a982b8fa2ea4 to start an election
I20240503 02:13:35.356860 13369 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:35.357707 10258 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7e966caa651e4e42890be5d79f851a1f"
dest_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4"
 from {username='slave'} at 127.0.145.3:58371
I20240503 02:13:35.357985 10258 raft_consensus.cc:482] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:35.358124 10258 raft_consensus.cc:3037] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:35.358484  9994 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:35.358954 13412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2448e12efbd44a90b1f8bce9519e73e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:35.352522  8101 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:35.361497 13382 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:35.361670 10258 raft_consensus.cc:504] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:35.365479 10258 leader_election.cc:288] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [CANDIDATE]: Term 2 election: Requested vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:35.366371  8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e966caa651e4e42890be5d79f851a1f" candidate_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:35.366667  8230 raft_consensus.cc:3032] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:35.366784  8230 raft_consensus.cc:729] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Running, Role: LEADER
I20240503 02:13:35.367146  8230 consensus_queue.cc:260] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:35.367596  8230 raft_consensus.cc:3037] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:35.368975  7823 catalog_manager.cc:5484] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 9 to 10, NON_VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) added. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } } }
I20240503 02:13:35.385277 13412 tablet_copy_client.cc:783] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:35.385666 13412 tablet_copy_client.cc:647] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:35.388165 13412 tablet_copy_client.cc:523] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:35.388928  8230 raft_consensus.cc:2454] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca05c9a8bee148dab8a2a982b8fa2ea4 in term 2.
I20240503 02:13:35.390105 10194 leader_election.cc:302] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, ca05c9a8bee148dab8a2a982b8fa2ea4; no voters: 
I20240503 02:13:35.390460 13382 raft_consensus.cc:2781] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:35.393654 13412 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
W20240503 02:13:35.408556  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:35.408931  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:35.409116  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:35.409575  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:35.426524 10197 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.145.5:44891, user_credentials={real_user=slave}} blocked reactor thread for 164711us
I20240503 02:13:35.428050  8361 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:35.433485 13514 ts_tablet_manager.cc:1875] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:35.441054 13382 raft_consensus.cc:686] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 LEADER]: Becoming Leader. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Running, Role: LEADER
I20240503 02:13:35.443116 13382 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } }
I20240503 02:13:35.445943  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e966caa651e4e42890be5d79f851a1f" candidate_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:35.446200  8361 raft_consensus.cc:3037] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:35.449997  8361 raft_consensus.cc:2454] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca05c9a8bee148dab8a2a982b8fa2ea4 in term 2.
I20240503 02:13:35.455770 10258 raft_consensus.cc:1261] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 02:13:35.456297 10197 leader_election.cc:334] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:35.477495 13018 consensus_queue.cc:1035] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:13:35.477702 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:35.478930 13382 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:35.479764 10258 raft_consensus.cc:2932] T 834aaba965c8443c862e98a070e36cac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:35.481058 13532 ts_tablet_manager.cc:902] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:13:35.490341 13514 ts_tablet_manager.cc:1888] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
W20240503 02:13:35.494931 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:35.495798 10241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:35.507714  7823 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: term changed from 1 to 2, leader changed from 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) to ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8). New cstate: current_term: 2 leader_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:35.508865 13412 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:35.509449 13412 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap complete.
I20240503 02:13:35.509755 13412 ts_tablet_manager.cc:1367] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.116s	user 0.055s	sys 0.031s
I20240503 02:13:35.510591 13412 raft_consensus.cc:348] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } }
I20240503 02:13:35.512035 13412 raft_consensus.cc:729] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: LEARNER
I20240503 02:13:35.517318 13514 log.cc:1196] T 834aaba965c8443c862e98a070e36cac P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/834aaba965c8443c862e98a070e36cac
I20240503 02:13:35.520500  7814 catalog_manager.cc:4830] TS 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): tablet 834aaba965c8443c862e98a070e36cac (table rebalance_test_table_8 [id=8da59f26450d4d25a86bc61be2943c37]) successfully deleted
W20240503 02:13:35.521674  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:35.512306 13412 consensus_queue.cc:260] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } }
I20240503 02:13:35.527395 13412 ts_tablet_manager.cc:1398] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.018s	user 0.003s	sys 0.000s
I20240503 02:13:35.528187  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session 9cbd8ddb1fc6479eb8cacbfd5b9bf86d-2448e12efbd44a90b1f8bce9519e73e0 received from {username='slave'} at 127.0.145.2:57675
I20240503 02:13:35.528379  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session 9cbd8ddb1fc6479eb8cacbfd5b9bf86d-2448e12efbd44a90b1f8bce9519e73e0 on tablet 2448e12efbd44a90b1f8bce9519e73e0 with peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d
I20240503 02:13:35.540804 13361 tablet_bootstrap.cc:492] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:35.594985 13361 tablet_bootstrap.cc:492] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap complete.
I20240503 02:13:35.598475 13361 ts_tablet_manager.cc:1367] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.247s	user 0.053s	sys 0.009s
I20240503 02:13:35.599596 13361 raft_consensus.cc:348] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } }
I20240503 02:13:35.600040 13361 raft_consensus.cc:729] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: LEARNER
I20240503 02:13:35.600358 13361 consensus_queue.cc:260] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } }
I20240503 02:13:35.602423 13361 ts_tablet_manager.cc:1398] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:13:35.602854 13532 tablet_copy_client.cc:308] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:13:35.603878  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 4af5884849ec4ac3891ebf0f08ee2cff from peer d986fcfb926e47baa7d2d5b99f697334 ({username='slave'} at 127.0.145.1:49559)
I20240503 02:13:35.604085  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 4af5884849ec4ac3891ebf0f08ee2cff from peer d986fcfb926e47baa7d2d5b99f697334 at {username='slave'} at 127.0.145.1:49559: session id = d986fcfb926e47baa7d2d5b99f697334-4af5884849ec4ac3891ebf0f08ee2cff
I20240503 02:13:35.608002  8361 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c23b06764ce54acb8388de94aea4cd4b"
dest_uuid: "8d7c037d50c241b484ee843b06313a64"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60054
I20240503 02:13:35.608291  8361 raft_consensus.cc:595] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:35.614951  8122 tablet_copy_source_session.cc:205] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:35.618419 13532 ts_tablet_manager.cc:1875] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:13:35.625164  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session 2942efc0ecb448d2ba2f6bca20fbb29c-45fe6ec58eed40f1898d4a2f5727cfa1 received from {username='slave'} at 127.0.145.3:57873
I20240503 02:13:35.625360  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session 2942efc0ecb448d2ba2f6bca20fbb29c-45fe6ec58eed40f1898d4a2f5727cfa1 on tablet 45fe6ec58eed40f1898d4a2f5727cfa1 with peer 2942efc0ecb448d2ba2f6bca20fbb29c
I20240503 02:13:35.666795  8361 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5ba1f505010b403c91d3870cee19952a"
dest_uuid: "8d7c037d50c241b484ee843b06313a64"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60062
I20240503 02:13:35.667075  8361 raft_consensus.cc:595] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:35.668777 13018 raft_consensus.cc:979] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64: : Instructing follower b85e85e631ab4e04a86bfe5a9f0aa512 to start an election
I20240503 02:13:35.668946 13018 raft_consensus.cc:1067] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Signalling peer b85e85e631ab4e04a86bfe5a9f0aa512 to start an election
I20240503 02:13:35.670059  9994 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c23b06764ce54acb8388de94aea4cd4b"
dest_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512"
 from {username='slave'} at 127.0.145.4:52991
I20240503 02:13:35.670297  9994 raft_consensus.cc:482] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:35.670449  9994 raft_consensus.cc:3037] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:35.672207 13532 ts_tablet_manager.cc:1888] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:13:35.672525 13532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af5884849ec4ac3891ebf0f08ee2cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:35.674813  9994 raft_consensus.cc:504] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:35.678133  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23b06764ce54acb8388de94aea4cd4b" candidate_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:35.678416  8361 raft_consensus.cc:3032] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:35.678503  8361 raft_consensus.cc:729] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Running, Role: LEADER
I20240503 02:13:35.678833  8361 consensus_queue.cc:260] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:35.679265  8361 raft_consensus.cc:3037] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:13:35.681828  9932 leader_election.cc:334] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:35.681955  7969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23b06764ce54acb8388de94aea4cd4b" candidate_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
I20240503 02:13:35.682163  7969 raft_consensus.cc:3037] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:35.682205  9994 leader_election.cc:288] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 election: Requested vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891)
I20240503 02:13:35.686426  7969 raft_consensus.cc:2454] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b85e85e631ab4e04a86bfe5a9f0aa512 in term 2.
I20240503 02:13:35.687274  8361 raft_consensus.cc:2454] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b85e85e631ab4e04a86bfe5a9f0aa512 in term 2.
I20240503 02:13:35.687886  9932 leader_election.cc:302] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, b85e85e631ab4e04a86bfe5a9f0aa512, d986fcfb926e47baa7d2d5b99f697334; no voters: 03f99420df704238ab300ad6e58d878c
I20240503 02:13:35.688318 13513 raft_consensus.cc:2781] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:35.688482 13513 raft_consensus.cc:686] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 LEADER]: Becoming Leader. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Running, Role: LEADER
I20240503 02:13:35.688819 13513 consensus_queue.cc:237] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } }
I20240503 02:13:35.698433 13532 tablet_copy_client.cc:783] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:35.698952 13532 tablet_copy_client.cc:647] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:35.701871 13532 tablet_copy_client.cc:523] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:35.708492  7828 catalog_manager.cc:5484] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: term changed from 1 to 2, leader changed from 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) to b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6). New cstate: current_term: 2 leader_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:35.709422 13532 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:13:35.752102  8230 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:13:35.752837 13273 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:13:35.754266 13412 ts_tablet_manager.cc:902] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Initiating tablet copy from peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377)
I20240503 02:13:35.768249 13273 raft_consensus.cc:482] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 03f99420df704238ab300ad6e58d878c)
I20240503 02:13:35.768498 13273 raft_consensus.cc:504] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:35.769742 13273 leader_election.cc:288] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891), d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
I20240503 02:13:35.770766  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:35.771333 10124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" is_pre_election: true
I20240503 02:13:35.772540  8038 leader_election.cc:302] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 8d7c037d50c241b484ee843b06313a64, d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:35.773840  8038 leader_election.cc:334] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:35.774451 13273 raft_consensus.cc:2726] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:13:35.776271  7969 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 2 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:13:35.777014 13273 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:13:35.822027 10258 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:13:35.824703 12774 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:13:35.826509 13016 raft_consensus.cc:482] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 03f99420df704238ab300ad6e58d878c)
I20240503 02:13:35.826795 13016 raft_consensus.cc:504] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:35.828054 13016 leader_election.cc:288] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891), d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
I20240503 02:13:35.829677 10124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" is_pre_election: true
I20240503 02:13:35.834810  8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:35.835105  8099 raft_consensus.cc:2454] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 2.
W20240503 02:13:35.835641  8298 leader_election.cc:334] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:35.835817  8298 leader_election.cc:302] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64; no voters: 03f99420df704238ab300ad6e58d878c, d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:35.836943 13016 raft_consensus.cc:2726] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:13:35.840854 13332 raft_consensus.cc:482] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 03f99420df704238ab300ad6e58d878c)
I20240503 02:13:35.841061 13332 raft_consensus.cc:504] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:35.842017 13332 leader_election.cc:288] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891)
I20240503 02:13:35.842975  8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:13:35.843269  8099 raft_consensus.cc:2454] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d826d4dae2aa4e05b2f64c9b17a1126f in term 2.
W20240503 02:13:35.851485 10064 leader_election.cc:334] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:35.852281  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:13:35.852577  8361 raft_consensus.cc:2454] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d826d4dae2aa4e05b2f64c9b17a1126f in term 2.
I20240503 02:13:35.855917 10064 leader_election.cc:302] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d826d4dae2aa4e05b2f64c9b17a1126f; no voters: 03f99420df704238ab300ad6e58d878c
I20240503 02:13:35.856670 13332 raft_consensus.cc:2781] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:13:35.856791 13332 raft_consensus.cc:482] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 03f99420df704238ab300ad6e58d878c)
I20240503 02:13:35.856874 13332 raft_consensus.cc:3037] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:35.860674 13332 raft_consensus.cc:504] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:35.864475  8099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:35.864733  8099 raft_consensus.cc:3037] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:35.867537  8099 raft_consensus.cc:2454] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d826d4dae2aa4e05b2f64c9b17a1126f in term 3.
I20240503 02:13:35.873102  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:35.873375  8361 raft_consensus.cc:3037] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:35.878106  8361 raft_consensus.cc:2454] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d826d4dae2aa4e05b2f64c9b17a1126f in term 3.
I20240503 02:13:35.878662 10064 leader_election.cc:302] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d826d4dae2aa4e05b2f64c9b17a1126f; no voters: 
I20240503 02:13:35.888866 13332 leader_election.cc:288] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [CANDIDATE]: Term 3 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891)
I20240503 02:13:35.890349 13332 raft_consensus.cc:2781] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:13:35.890591 13332 raft_consensus.cc:686] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 3 LEADER]: Becoming Leader. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Running, Role: LEADER
I20240503 02:13:35.891039 13332 consensus_queue.cc:237] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:35.903152 13369 raft_consensus.cc:979] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64: : Instructing follower d986fcfb926e47baa7d2d5b99f697334 to start an election
I20240503 02:13:35.903337 13369 raft_consensus.cc:1067] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Signalling peer d986fcfb926e47baa7d2d5b99f697334 to start an election
I20240503 02:13:35.904542  7968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ba1f505010b403c91d3870cee19952a"
dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
 from {username='slave'} at 127.0.145.4:51979
I20240503 02:13:35.904860  7968 raft_consensus.cc:482] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:35.905006  7968 raft_consensus.cc:3037] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:35.910123  7968 raft_consensus.cc:504] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:35.920210  7968 leader_election.cc:288] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 2 election: Requested vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
I20240503 02:13:35.920354  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1f505010b403c91d3870cee19952a" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:35.920617  8232 raft_consensus.cc:3037] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:35.921129  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1f505010b403c91d3870cee19952a" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:13:35.921352  8361 raft_consensus.cc:3032] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:35.921440  8361 raft_consensus.cc:729] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Running, Role: LEADER
I20240503 02:13:35.921887 10124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1f505010b403c91d3870cee19952a" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f"
I20240503 02:13:35.921825  8361 consensus_queue.cc:260] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:35.922106 10124 raft_consensus.cc:3037] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:35.922294  8361 raft_consensus.cc:3037] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:13:35.924006 10064 leader_election.cc:334] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:35.926815  8232 raft_consensus.cc:2454] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 2.
I20240503 02:13:35.928166 10124 raft_consensus.cc:2454] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 2.
I20240503 02:13:35.928761  7904 leader_election.cc:302] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, d826d4dae2aa4e05b2f64c9b17a1126f, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:35.929154 12411 raft_consensus.cc:2781] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:35.929172  8361 raft_consensus.cc:2454] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 2.
I20240503 02:13:35.934831  7828 catalog_manager.cc:5484] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: term changed from 2 to 3, leader changed from 03f99420df704238ab300ad6e58d878c (127.0.145.5) to d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7). New cstate: current_term: 3 leader_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 02:13:35.949728 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:35.951220 13607 log.cc:924] Time spent T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20240503 02:13:35.954563 13412 tablet_copy_client.cc:308] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.4:35377
I20240503 02:13:35.955464  8382 tablet_copy_service.cc:140] P 8d7c037d50c241b484ee843b06313a64: Received BeginTabletCopySession request for tablet 80b4e6aa20f64473be8bcbf47baf66e4 from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d ({username='slave'} at 127.0.145.2:46225)
I20240503 02:13:35.955719  8382 tablet_copy_service.cc:161] P 8d7c037d50c241b484ee843b06313a64: Beginning new tablet copy session on tablet 80b4e6aa20f64473be8bcbf47baf66e4 from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d at {username='slave'} at 127.0.145.2:46225: session id = 9cbd8ddb1fc6479eb8cacbfd5b9bf86d-80b4e6aa20f64473be8bcbf47baf66e4
W20240503 02:13:35.960996  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:35.964489  8382 tablet_copy_source_session.cc:205] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:35.967694 13412 ts_tablet_manager.cc:1875] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:13:35.970994 12411 raft_consensus.cc:686] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 2 LEADER]: Becoming Leader. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Running, Role: LEADER
I20240503 02:13:35.971493 12411 consensus_queue.cc:237] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:35.982445 13412 ts_tablet_manager.cc:1888] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240503 02:13:35.983426  8099 raft_consensus.cc:1203] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:13:35.988065  7827 catalog_manager.cc:5484] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: term changed from 1 to 2, leader changed from 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) to d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1). New cstate: current_term: 2 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:35.988425 13412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b4e6aa20f64473be8bcbf47baf66e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:35.992756 13412 tablet_copy_client.cc:783] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:35.993065 13412 tablet_copy_client.cc:647] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:35.996143 13412 tablet_copy_client.cc:523] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:36.000295 13412 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap starting.
I20240503 02:13:36.013000  8230 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:13:36.014031 13382 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:13:36.029984  8361 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:13:36.031236 13524 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:13:36.045723  8230 raft_consensus.cc:1203] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240503 02:13:36.075961  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:36.110707 13412 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:36.111281 13412 tablet_bootstrap.cc:492] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Bootstrap complete.
I20240503 02:13:36.111562 13412 ts_tablet_manager.cc:1367] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent bootstrapping tablet: real 0.111s	user 0.071s	sys 0.003s
I20240503 02:13:36.112342 13412 raft_consensus.cc:348] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } }
I20240503 02:13:36.112562 13412 raft_consensus.cc:729] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Initialized, Role: LEARNER
I20240503 02:13:36.112793 13412 consensus_queue.cc:260] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: true } }
I20240503 02:13:36.113330 13412 ts_tablet_manager.cc:1398] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:13:36.116561  8382 tablet_copy_service.cc:342] P 8d7c037d50c241b484ee843b06313a64: Request end of tablet copy session 9cbd8ddb1fc6479eb8cacbfd5b9bf86d-80b4e6aa20f64473be8bcbf47baf66e4 received from {username='slave'} at 127.0.145.2:46225
I20240503 02:13:36.116710  8382 tablet_copy_service.cc:433] P 8d7c037d50c241b484ee843b06313a64: ending tablet copy session 9cbd8ddb1fc6479eb8cacbfd5b9bf86d-80b4e6aa20f64473be8bcbf47baf66e4 on tablet 80b4e6aa20f64473be8bcbf47baf66e4 with peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d
I20240503 02:13:36.122305  7969 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 2 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:36.123301 13513 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:36.142506 13532 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:36.142962 13532 tablet_bootstrap.cc:492] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: Bootstrap complete.
I20240503 02:13:36.143239 13532 ts_tablet_manager.cc:1367] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.436s	user 0.067s	sys 0.002s
I20240503 02:13:36.143954 13532 raft_consensus.cc:348] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } }
I20240503 02:13:36.144191 13532 raft_consensus.cc:729] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: LEARNER
I20240503 02:13:36.147215 13532 consensus_queue.cc:260] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } }
I20240503 02:13:36.148706 13532 ts_tablet_manager.cc:1398] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:13:36.151691  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session d986fcfb926e47baa7d2d5b99f697334-4af5884849ec4ac3891ebf0f08ee2cff received from {username='slave'} at 127.0.145.1:49559
I20240503 02:13:36.151832  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session d986fcfb926e47baa7d2d5b99f697334-4af5884849ec4ac3891ebf0f08ee2cff on tablet 4af5884849ec4ac3891ebf0f08ee2cff with peer d986fcfb926e47baa7d2d5b99f697334
I20240503 02:13:36.249191  8361 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:36.250771 13513 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:36.257581  8099 consensus_queue.cc:237] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } }
I20240503 02:13:36.260424 10258 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:13:36.261344 13053 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:13:36.264667  7969 raft_consensus.cc:1261] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 2 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:13:36.265491 13053 consensus_queue.cc:1035] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:13:36.272541 12774 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:36.277664 10258 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:36.292810  7969 raft_consensus.cc:1203] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
I20240503 02:13:36.299356  7969 raft_consensus.cc:2932] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:36.301033  8099 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240503 02:13:36.303973 13332 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:36.309204 10108 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:36.316800  7830 catalog_manager.cc:5484] T 2d69c71d9dfd47a1a2429e7caf5e33ac P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 11 to 13, VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) evicted. New cstate: current_term: 2 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:36.321208  8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac with delete_type TABLET_DATA_TOMBSTONED (TS 2942efc0ecb448d2ba2f6bca20fbb29c not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:35426
I20240503 02:13:36.321380  8099 raft_consensus.cc:1203] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240503 02:13:36.332944  8361 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240503 02:13:36.334388 13332 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:36.342566  7950 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:36.344201  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d69c71d9dfd47a1a2429e7caf5e33ac with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:36.370854  8230 raft_consensus.cc:1261] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:36.373011 13623 consensus_queue.cc:1035] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:36.378322 13690 tablet_replica.cc:330] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: stopping tablet replica
I20240503 02:13:36.380056 13690 raft_consensus.cc:2229] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:36.381275 13690 raft_consensus.cc:2258] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:36.383862 13690 ts_tablet_manager.cc:1875] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:36.384088 10241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:36.410341 13690 ts_tablet_manager.cc:1888] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240503 02:13:36.418751  8362 raft_consensus.cc:1261] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:36.420219 13623 consensus_queue.cc:1035] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:13:36.426795  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:36.436110 13690 log.cc:1196] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/2d69c71d9dfd47a1a2429e7caf5e33ac
I20240503 02:13:36.450681  7814 catalog_manager.cc:4830] TS 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): tablet 2d69c71d9dfd47a1a2429e7caf5e33ac (table rebalance_test_table_0 [id=e33f513f068647b9921daa3a38cb3c30]) successfully deleted
I20240503 02:13:36.465648  8081 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:13:36.475345 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:36.489343 13273 raft_consensus.cc:1050] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c to VOTER
I20240503 02:13:36.490361 13273 consensus_queue.cc:237] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:13:36.504374 10124 raft_consensus.cc:1261] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:36.505277 12411 consensus_queue.cc:1035] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:36.535271  8211 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:36.539335  8341 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:36.578923 13486 raft_consensus.cc:1050] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to VOTER
I20240503 02:13:36.580261 13486 consensus_queue.cc:237] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } }
I20240503 02:13:36.596048  9976 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:36.612718  8232 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:36.613505 12774 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:36.625607  7969 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:36.634364  9994 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:36.635450 12774 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:36.636140 12774 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:36.664069  8100 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:36.681216 12774 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
W20240503 02:13:36.681715  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:36.682685  8232 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:13:36.705085  9994 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:13:36.710888 13711 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:36.716104  7823 catalog_manager.cc:5484] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 7 to 8, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:13:36.725241  7969 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
W20240503 02:13:36.744450  8167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.2:35309, user_credentials={real_user=slave}} blocked reactor thread for 79741.3us
I20240503 02:13:36.746490  8362 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:36.748649 13736 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240503 02:13:36.784161 13486 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:36.792338  8362 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:36.801213  7828 catalog_manager.cc:5484] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 7 to 8, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:36.805904  8100 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:36.814146 13018 raft_consensus.cc:1050] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64: attempting to promote NON_VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to VOTER
I20240503 02:13:36.815196 13018 consensus_queue.cc:237] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } }
I20240503 02:13:36.824352  8099 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEARNER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 02:13:36.824532  8298 consensus_peers.cc:485] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:36.825129 13016 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:36.827952  9994 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:13:36.828668 13016 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:13:36.854619 13016 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:36.859006  9995 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:36.868794  7826 catalog_manager.cc:5484] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 10 to 11, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:36.873088 10309 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240503 02:13:13.109233 10314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:13.109680 10314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:13.122684 10314 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:16.164177 10314 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:13:17.311765 10829 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.0.145.62:36865, user_credentials={real_user=slave}} blocked reactor thread for 49122.2us
W20240503 02:13:17.975188 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:17.975572 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:17.975718 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:17.975849 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:17.976133 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:17.976433 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b1c050f6f652453f9abd0ce2412df9e9: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:17.976742 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:17.976919 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7d72eaa895849fcb02235c1b6e8d08f: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:17.977105 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8da59f26450d4d25a86bc61be2943c37: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:17.977334 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:17.977468 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b1c050f6f652453f9abd0ce2412df9e9: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:17.977612 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:17.977728 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:17.977869 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server 03f99420df704238ab300ad6e58d878c
I20240503 02:13:18.130211 10314 rebalancer_tool.cc:1074] tablet 0f44b2b53de84a87a8757c0d00bfed37: '8d7c037d50c241b484ee843b06313a64' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move scheduled
I20240503 02:13:18.446681 10314 rebalancer_tool.cc:1074] tablet f14607425da84bb4b162a5789daa844c: 'd986fcfb926e47baa7d2d5b99f697334' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move scheduled
I20240503 02:13:18.729745 10314 rebalancer_tool.cc:1074] tablet 5ea56405515a40f785726bd58ec3d26a: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move scheduled
I20240503 02:13:19.012792 10314 rebalancer_tool.cc:1074] tablet aeaee03abc644807b8278cfb78e4f02b: '2942efc0ecb448d2ba2f6bca20fbb29c' -> '03f99420df704238ab300ad6e58d878c' move scheduled
I20240503 02:13:19.253890 10314 rebalancer_tool.cc:1074] tablet de9c2a3396864696af88be1ca12ee0f1: '8d7c037d50c241b484ee843b06313a64' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move scheduled
I20240503 02:13:19.468442 10314 rebalancer_tool.cc:1074] tablet d3d4b912095244de816d3089a31b8649: 'd986fcfb926e47baa7d2d5b99f697334' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move scheduled
I20240503 02:13:19.813182 10314 rebalancer_tool.cc:1074] tablet 4af5884849ec4ac3891ebf0f08ee2cff: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move scheduled
I20240503 02:13:20.622776 10314 rebalancer_tool.cc:1074] tablet 4f2cedba9f9a4e709623fceeffcfd095: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move scheduled
I20240503 02:13:20.776398 10314 rebalancer_tool.cc:1084] tablet ceae1c7c45a34b24badbbe23cb9e0176: 'd986fcfb926e47baa7d2d5b99f697334' -> '03f99420df704238ab300ad6e58d878c' move ignored: Illegal state: tablet ceae1c7c45a34b24badbbe23cb9e0176: replica 03f99420df704238ab300ad6e58d878c is already present
I20240503 02:13:20.778903 10314 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:23.066434 11634 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.0.145.62:36865, user_credentials={real_user=slave}} blocked reactor thread for 48726.1us
W20240503 02:13:24.645578 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:24.645967 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7d72eaa895849fcb02235c1b6e8d08f: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:24.646193 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:24.646328 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:24.648756 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:24.648977 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8da59f26450d4d25a86bc61be2943c37: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:24.649097 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:24.649222 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:24.649331 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:24.649459 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:24.955399 10314 rebalancer_tool.cc:1074] tablet b64f66ace86e42f3ac213ba85bcab584: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '' move scheduled
I20240503 02:13:25.354360 10314 rebalancer_tool.cc:1074] tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298: '8d7c037d50c241b484ee843b06313a64' -> '03f99420df704238ab300ad6e58d878c' move scheduled
I20240503 02:13:25.696499 10314 rebalancer_tool.cc:1074] tablet 402de43677c44deab8793da1895d4fb2: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move scheduled
I20240503 02:13:25.932528 10314 rebalancer_tool.cc:1084] tablet 834aaba965c8443c862e98a070e36cac: 'd986fcfb926e47baa7d2d5b99f697334' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move ignored: Illegal state: tablet 834aaba965c8443c862e98a070e36cac: replica ca05c9a8bee148dab8a2a982b8fa2ea4 is already present
I20240503 02:13:25.932678 10314 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:29.328666 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7d72eaa895849fcb02235c1b6e8d08f: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:29.328797 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:29.328897 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8da59f26450d4d25a86bc61be2943c37: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:29.329030 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server 2942efc0ecb448d2ba2f6bca20fbb29c
W20240503 02:13:29.329108 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:29.329249 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:29.329337 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:29.329401 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:29.329485 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8da59f26450d4d25a86bc61be2943c37: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:29.329555 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:29.657469 10314 rebalancer_tool.cc:1074] tablet 2d69c71d9dfd47a1a2429e7caf5e33ac: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '03f99420df704238ab300ad6e58d878c' move scheduled
I20240503 02:13:30.031461 10314 rebalancer_tool.cc:1074] tablet 6043da7d2e584af885ff6a5bf8f06912: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move scheduled
I20240503 02:13:30.365207 10314 rebalancer_tool.cc:1074] tablet c45c5a14b85447c78fe2a27be4392c88: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '8d7c037d50c241b484ee843b06313a64' move scheduled
I20240503 02:13:30.514328 10314 rebalancer_tool.cc:1084] tablet 8a8affdde4d8417f95f0585dd01e7d59: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move ignored: Illegal state: tablet 8a8affdde4d8417f95f0585dd01e7d59: replica d826d4dae2aa4e05b2f64c9b17a1126f is already present
I20240503 02:13:30.514578 10314 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:33.013643 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 03f99420df704238ab300ad6e58d878c to server 2942efc0ecb448d2ba2f6bca20fbb29c
W20240503 02:13:33.013940 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server ca05c9a8bee148dab8a2a982b8fa2ea4 to server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d
W20240503 02:13:33.015619 10314 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server 2942efc0ecb448d2ba2f6bca20fbb29c
I20240503 02:13:33.320639 10314 rebalancer_tool.cc:1074] tablet 7e966caa651e4e42890be5d79f851a1f: 'd986fcfb926e47baa7d2d5b99f697334' -> '' move scheduled
I20240503 02:13:33.736198 10314 rebalancer_tool.cc:1084] tablet 834aaba965c8443c862e98a070e36cac: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'd986fcfb926e47baa7d2d5b99f697334' move ignored: Illegal state: tablet 834aaba965c8443c862e98a070e36cac: replica d986fcfb926e47baa7d2d5b99f697334 is already present
I20240503 02:13:33.736405 10314 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:34.989077 13329 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:13:34.539115 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.145.5:44891 (local address 127.0.0.1:52366)
0503 02:13:34.559472 (+ 20357us) negotiation.cc:107] Waiting for socket to connect
0503 02:13:34.559502 (+    30us) client_negotiation.cc:174] Beginning negotiation
0503 02:13:34.559711 (+   209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:13:34.987561 (+427850us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":20160,"thread_start_us":21503,"threads_started":1}
Illegal state: tablet server 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): unacceptable health status UNAVAILABLE

I20240503 02:13:36.879159  8362 consensus_queue.cc:237] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } }
I20240503 02:13:36.883662  9994 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:13:36.884490 13016 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:13:36.871031  8100 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:36.889701 13016 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:36.895555  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80b4e6aa20f64473be8bcbf47baf66e4 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 02:13:36.897331  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:36.897959  7826 catalog_manager.cc:5484] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 11 to 12, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 1 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:36.912945  9995 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:36.914484 13053 raft_consensus.cc:1050] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER d986fcfb926e47baa7d2d5b99f697334 to VOTER
I20240503 02:13:36.915674 13053 consensus_queue.cc:237] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } }
W20240503 02:13:36.919747  7817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:36.962740  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 80b4e6aa20f64473be8bcbf47baf66e4: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:36.964696  7969 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:13:36.965688 13273 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 02:13:36.972064  8038 consensus_peers.cc:485] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:36.972872  9995 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:13:36.973582 12774 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:13:36.990145  7968 consensus_queue.cc:237] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:13:36.993315  8232 raft_consensus.cc:1261] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:36.994727 12411 consensus_queue.cc:1035] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:36.998759 13732 raft_consensus.cc:2932] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:36.999531  8099 raft_consensus.cc:1261] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:13:37.003371  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5ba1f505010b403c91d3870cee19952a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:37.003765 13018 consensus_queue.cc:1035] T 80b4e6aa20f64473be8bcbf47baf66e4 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:13:37.005956  8232 raft_consensus.cc:2932] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:37.006122  8099 raft_consensus.cc:2932] T 80b4e6aa20f64473be8bcbf47baf66e4 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:13:37.011655 10124 raft_consensus.cc:1261] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:37.013161  7828 catalog_manager.cc:5484] T 5ba1f505010b403c91d3870cee19952a P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 5 to 7, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New cstate: current_term: 2 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:37.013746 13704 consensus_queue.cc:1035] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:37.016093 10124 raft_consensus.cc:2932] T 5ba1f505010b403c91d3870cee19952a P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:13:37.017596  8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ba1f505010b403c91d3870cee19952a with delete_type TABLET_DATA_TOMBSTONED (TS 8d7c037d50c241b484ee843b06313a64 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37080
I20240503 02:13:37.046990  8232 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:13:37.048349 13273 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:13:37.048935 13273 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:37.056721  7968 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:37.059787 13779 tablet_replica.cc:330] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64: stopping tablet replica
I20240503 02:13:37.060016 13779 raft_consensus.cc:2229] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:37.060246 13779 raft_consensus.cc:2258] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:37.061544 13779 ts_tablet_manager.cc:1875] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:37.068089  7823 catalog_manager.cc:5484] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 11 to 12, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:37.078992  9995 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:37.091574  8232 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:37.106223 13779 ts_tablet_manager.cc:1888] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:13:37.106479 13779 log.cc:1196] T 5ba1f505010b403c91d3870cee19952a P 8d7c037d50c241b484ee843b06313a64: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/wals/5ba1f505010b403c91d3870cee19952a
I20240503 02:13:37.107507  7817 catalog_manager.cc:4830] TS 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): tablet 5ba1f505010b403c91d3870cee19952a (table rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]) successfully deleted
I20240503 02:13:37.140619  8211 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:37.216009  8341 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:37.216446  9976 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:37.221982  8081 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:37.271284 10108 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:37.306107  7950 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:37.341077 10241 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:13:37.353157  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:37.452093 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:13:37.534101  8100 consensus_queue.cc:237] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } }
I20240503 02:13:37.538812  9994 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:13:37.538990  8232 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:13:37.539472 13775 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:13:37.539700 13273 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:13:37.546334 13775 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:37.552485  8232 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:37.560750  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4af5884849ec4ac3891ebf0f08ee2cff with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:37.560649  7826 catalog_manager.cc:5484] T 4af5884849ec4ac3891ebf0f08ee2cff P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 12 to 13, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:37.565392  9994 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
W20240503 02:13:37.575459  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:13:37.606113  7968 raft_consensus.cc:1261] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240503 02:13:37.606792 13775 consensus_queue.cc:1035] T 4af5884849ec4ac3891ebf0f08ee2cff P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:13:37.622663  7968 raft_consensus.cc:2932] T 4af5884849ec4ac3891ebf0f08ee2cff P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: false } } }
I20240503 02:13:37.638857 10312 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240503 02:13:13.107080 10316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:13.107506 10316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:13.120368 10316 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:16.064385 10316 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:13:17.852105 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:17.852496 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:17.852834 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b1c050f6f652453f9abd0ce2412df9e9: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:17.852982 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:17.853120 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:17.853260 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:17.853365 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:17.853772 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7d72eaa895849fcb02235c1b6e8d08f: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:17.853926 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8da59f26450d4d25a86bc61be2943c37: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:17.854048 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:17.854154 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b1c050f6f652453f9abd0ce2412df9e9: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:17.854288 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:17.854387 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:17.854542 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:18.038594 10316 rebalancer_tool.cc:1074] tablet 0f44b2b53de84a87a8757c0d00bfed37: '8d7c037d50c241b484ee843b06313a64' -> '03f99420df704238ab300ad6e58d878c' move scheduled
I20240503 02:13:18.196769 10316 rebalancer_tool.cc:1074] tablet 80b4e6aa20f64473be8bcbf47baf66e4: 'd986fcfb926e47baa7d2d5b99f697334' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move scheduled
I20240503 02:13:18.418876 10316 rebalancer_tool.cc:1074] tablet 4f2cedba9f9a4e709623fceeffcfd095: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move scheduled
I20240503 02:13:18.601464 10316 rebalancer_tool.cc:1074] tablet b64f66ace86e42f3ac213ba85bcab584: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move scheduled
I20240503 02:13:18.934311 10316 rebalancer_tool.cc:1074] tablet 72e2bb75da654e8ab45bba8ac077f28c: '8d7c037d50c241b484ee843b06313a64' -> '03f99420df704238ab300ad6e58d878c' move scheduled
I20240503 02:13:19.280365 10316 rebalancer_tool.cc:1074] tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298: 'd986fcfb926e47baa7d2d5b99f697334' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move scheduled
I20240503 02:13:19.640270 10316 rebalancer_tool.cc:1074] tablet 32e527fa9b2f441cb5668435681b4ff6: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move scheduled
I20240503 02:13:19.998773 10316 rebalancer_tool.cc:1074] tablet 3dbfc7f12fc142f98b1d1c026fd49f5a: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move scheduled
I20240503 02:13:20.910575 10316 rebalancer_tool.cc:1074] tablet f231fe5dfd9e4c87b90bad75818a7060: '8d7c037d50c241b484ee843b06313a64' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move scheduled
I20240503 02:13:21.142807 10316 rebalancer_tool.cc:1084] tablet ceae1c7c45a34b24badbbe23cb9e0176: 'd986fcfb926e47baa7d2d5b99f697334' -> '03f99420df704238ab300ad6e58d878c' move ignored: Illegal state: tablet ceae1c7c45a34b24badbbe23cb9e0176: replica 03f99420df704238ab300ad6e58d878c is already present
I20240503 02:13:21.142968 10316 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:23.093508 11621 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.0.145.62:36865, user_credentials={real_user=slave}} blocked reactor thread for 52914.2us
W20240503 02:13:24.539458 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:24.539634 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:24.539741 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:24.539860 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:24.539961 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:24.540084 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:24.540292 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:24.540364 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8da59f26450d4d25a86bc61be2943c37: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:24.540534 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b1c050f6f652453f9abd0ce2412df9e9: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:24.540620 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:24.692296 10316 rebalancer_tool.cc:1084] tablet de9c2a3396864696af88be1ca12ee0f1: '2942efc0ecb448d2ba2f6bca20fbb29c' -> '' move ignored: Not found: Peer with uuid 2942efc0ecb448d2ba2f6bca20fbb29c not found in consensus config
I20240503 02:13:24.692482 10316 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:28.192808 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:28.193279 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:28.194010 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:28.197558 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:28.197971 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:28.204161 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:28.209087 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:28.209589 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8da59f26450d4d25a86bc61be2943c37: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:28.307197 10316 rebalancer_tool.cc:1074] tablet aeaee03abc644807b8278cfb78e4f02b: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '' move scheduled
I20240503 02:13:28.766724 10316 rebalancer_tool.cc:1074] tablet 4af5884849ec4ac3891ebf0f08ee2cff: 'd986fcfb926e47baa7d2d5b99f697334' -> '' move scheduled
I20240503 02:13:29.033468 10316 rebalancer_tool.cc:1074] tablet 007b59de142c4b95966494ba097b1a09: '2942efc0ecb448d2ba2f6bca20fbb29c' -> '03f99420df704238ab300ad6e58d878c' move scheduled
I20240503 02:13:29.147872 10316 rebalancer_tool.cc:1084] tablet 5ea56405515a40f785726bd58ec3d26a: '8d7c037d50c241b484ee843b06313a64' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move ignored: Illegal state: tablet 5ea56405515a40f785726bd58ec3d26a: replica ca05c9a8bee148dab8a2a982b8fa2ea4 is already present
I20240503 02:13:29.148030 10316 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:31.662278 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server 2942efc0ecb448d2ba2f6bca20fbb29c
W20240503 02:13:31.662545 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:31.662698 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7d72eaa895849fcb02235c1b6e8d08f: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:31.662840 10316 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:31.869895 10316 rebalancer_tool.cc:1074] tablet cab70b62bdc747a9b653bffde680f37e: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move scheduled
I20240503 02:13:32.469756 10316 rebalancer_tool.cc:1074] tablet 2d69c71d9dfd47a1a2429e7caf5e33ac: '03f99420df704238ab300ad6e58d878c' -> '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' move scheduled
I20240503 02:13:32.682411 10316 rebalancer_tool.cc:1074] tablet 834aaba965c8443c862e98a070e36cac: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'd986fcfb926e47baa7d2d5b99f697334' move scheduled
I20240503 02:13:33.189273 10316 rebalancer_tool.cc:1136] tablet 007b59de142c4b95966494ba097b1a09: '2942efc0ecb448d2ba2f6bca20fbb29c' -> '03f99420df704238ab300ad6e58d878c' move completed: OK
I20240503 02:13:33.267889 10316 rebalancer_tool.cc:1136] tablet 4af5884849ec4ac3891ebf0f08ee2cff: 'd986fcfb926e47baa7d2d5b99f697334' -> '' move completed: OK
I20240503 02:13:33.570025 10316 rebalancer_tool.cc:1136] tablet f231fe5dfd9e4c87b90bad75818a7060: '8d7c037d50c241b484ee843b06313a64' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move completed: OK
I20240503 02:13:33.715015 10316 rebalancer_tool.cc:1136] tablet 80b4e6aa20f64473be8bcbf47baf66e4: 'd986fcfb926e47baa7d2d5b99f697334' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move completed: OK
I20240503 02:13:33.839103 10316 rebalancer_tool.cc:1136] tablet 0f44b2b53de84a87a8757c0d00bfed37: '8d7c037d50c241b484ee843b06313a64' -> '03f99420df704238ab300ad6e58d878c' move completed: OK
I20240503 02:13:33.887910 10316 rebalancer_tool.cc:1136] tablet 4f2cedba9f9a4e709623fceeffcfd095: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move completed: OK
I20240503 02:13:34.160719 10316 rebalancer_tool.cc:1136] tablet 3dbfc7f12fc142f98b1d1c026fd49f5a: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move completed: OK
I20240503 02:13:34.992944 10316 rebalancer_tool.cc:1136] tablet 72e2bb75da654e8ab45bba8ac077f28c: '8d7c037d50c241b484ee843b06313a64' -> '03f99420df704238ab300ad6e58d878c' move completed: OK
I20240503 02:13:35.003898 10316 rebalancer_tool.cc:1136] tablet 32e527fa9b2f441cb5668435681b4ff6: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move completed: OK
I20240503 02:13:35.050151 10316 rebalancer_tool.cc:1136] tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298: 'd986fcfb926e47baa7d2d5b99f697334' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move completed: OK
I20240503 02:13:35.135324 10316 rebalancer_tool.cc:1084] tablet 2448e12efbd44a90b1f8bce9519e73e0: '8d7c037d50c241b484ee843b06313a64' -> '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' move ignored: Illegal state: tablet 2448e12efbd44a90b1f8bce9519e73e0: replica 9cbd8ddb1fc6479eb8cacbfd5b9bf86d is already present
I20240503 02:13:35.135509 10316 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): unacceptable health status UNAVAILABLE

W20240503 02:13:37.661700 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:13:37.710705  8100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "45fe6ec58eed40f1898d4a2f5727cfa1"
dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60080
I20240503 02:13:37.710923  8100 raft_consensus.cc:595] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:37.836989  8100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "45fe6ec58eed40f1898d4a2f5727cfa1"
dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60090
I20240503 02:13:37.837213  8100 raft_consensus.cc:595] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Received request to transfer leadership
W20240503 02:13:37.839064  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:37.875690 13053 raft_consensus.cc:979] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: : Instructing follower d986fcfb926e47baa7d2d5b99f697334 to start an election
I20240503 02:13:37.875883 13053 raft_consensus.cc:1067] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Signalling peer d986fcfb926e47baa7d2d5b99f697334 to start an election
I20240503 02:13:37.876653  7968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "45fe6ec58eed40f1898d4a2f5727cfa1"
dest_uuid: "d986fcfb926e47baa7d2d5b99f697334"
 from {username='slave'} at 127.0.145.2:57601
I20240503 02:13:37.876916  7968 raft_consensus.cc:482] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:37.877060  7968 raft_consensus.cc:3037] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:37.881125  7968 raft_consensus.cc:504] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:13:37.884140  8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45fe6ec58eed40f1898d4a2f5727cfa1" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:13:37.884373  8100 raft_consensus.cc:3032] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:37.884469  8100 raft_consensus.cc:729] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:13:37.884764  8100 consensus_queue.cc:260] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:13:37.885216  8100 raft_consensus.cc:3037] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:37.888409  8100 raft_consensus.cc:2454] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 2.
I20240503 02:13:37.889367  7968 leader_election.cc:288] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 2 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:13:37.889791  8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45fe6ec58eed40f1898d4a2f5727cfa1" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:13:37.890061  8230 raft_consensus.cc:3037] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:37.890074  9995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45fe6ec58eed40f1898d4a2f5727cfa1" candidate_uuid: "d986fcfb926e47baa7d2d5b99f697334" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512"
I20240503 02:13:37.890295  9995 raft_consensus.cc:3037] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:37.893771  9995 raft_consensus.cc:2454] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 2.
I20240503 02:13:37.894398  8230 raft_consensus.cc:2454] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d986fcfb926e47baa7d2d5b99f697334 in term 2.
I20240503 02:13:37.894631  7907 leader_election.cc:302] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, b85e85e631ab4e04a86bfe5a9f0aa512, d986fcfb926e47baa7d2d5b99f697334; no voters: 
I20240503 02:13:37.895145 13704 raft_consensus.cc:2781] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:37.895375 13704 raft_consensus.cc:686] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 2 LEADER]: Becoming Leader. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Running, Role: LEADER
I20240503 02:13:37.895774 13704 consensus_queue.cc:237] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:13:37.897922 10311 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240503 02:13:13.387084 10317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:13.387575 10317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:13.422052 10317 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:16.909838 10317 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:13:18.568388 10844 outbound_call.cc:322] RPC callback for RPC call kudu.server.GenericService.GetStatus -> {remote=127.0.145.7:39415, user_credentials={real_user=slave}} blocked reactor thread for 44195.4us
W20240503 02:13:19.343469 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:19.350873 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b1c050f6f652453f9abd0ce2412df9e9: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:19.351593 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:19.352088 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:19.352747 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b1c050f6f652453f9abd0ce2412df9e9: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:19.353514 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:19.353906 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:19.354192 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:19.354481 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:19.354928 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7d72eaa895849fcb02235c1b6e8d08f: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:19.355304 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:19.355569 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:19.355846 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8da59f26450d4d25a86bc61be2943c37: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:19.356155 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:19.356446 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:19.356720 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:19.358374 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b1c050f6f652453f9abd0ce2412df9e9: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:19.358752 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:19.718142 10317 rebalancer_tool.cc:1074] tablet cab70b62bdc747a9b653bffde680f37e: '8d7c037d50c241b484ee843b06313a64' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move scheduled
I20240503 02:13:19.843348 10317 rebalancer_tool.cc:1074] tablet aeaee03abc644807b8278cfb78e4f02b: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move scheduled
I20240503 02:13:20.174250 10317 rebalancer_tool.cc:1074] tablet ceae1c7c45a34b24badbbe23cb9e0176: 'd986fcfb926e47baa7d2d5b99f697334' -> '03f99420df704238ab300ad6e58d878c' move scheduled
I20240503 02:13:20.535601 10317 rebalancer_tool.cc:1074] tablet 7e966caa651e4e42890be5d79f851a1f: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move scheduled
I20240503 02:13:20.866063 10317 rebalancer_tool.cc:1074] tablet c23b06764ce54acb8388de94aea4cd4b: '8d7c037d50c241b484ee843b06313a64' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move scheduled
I20240503 02:13:21.253710 10317 rebalancer_tool.cc:1084] tablet 72e2bb75da654e8ab45bba8ac077f28c: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move ignored: Illegal state: tablet 72e2bb75da654e8ab45bba8ac077f28c: replica d826d4dae2aa4e05b2f64c9b17a1126f is already present
I20240503 02:13:21.253854 10317 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:22.588959 11636 meta_cache.cc:1330] Time spent looking up entry by ID: real 0.087s	user 0.000s	sys 0.000s
W20240503 02:13:24.178421 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:24.178790 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:24.178933 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:24.179044 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:24.179231 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:24.179409 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8da59f26450d4d25a86bc61be2943c37: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:24.179553 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7d72eaa895849fcb02235c1b6e8d08f: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:24.179646 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:24.179771 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:24.179859 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b1c050f6f652453f9abd0ce2412df9e9: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:24.440761 10317 rebalancer_tool.cc:1074] tablet de9c2a3396864696af88be1ca12ee0f1: '2942efc0ecb448d2ba2f6bca20fbb29c' -> '' move scheduled
I20240503 02:13:24.713107 10317 rebalancer_tool.cc:1074] tablet 2448e12efbd44a90b1f8bce9519e73e0: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '03f99420df704238ab300ad6e58d878c' move scheduled
I20240503 02:13:24.896077 10317 rebalancer_tool.cc:1074] tablet 834aaba965c8443c862e98a070e36cac: 'd986fcfb926e47baa7d2d5b99f697334' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move scheduled
I20240503 02:13:25.130156 10317 rebalancer_tool.cc:1074] tablet ecb5c78a22a74728ab40f1da8f90388c: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move scheduled
I20240503 02:13:25.320325 10317 rebalancer_tool.cc:1074] tablet b64f66ace86e42f3ac213ba85bcab584: '2942efc0ecb448d2ba2f6bca20fbb29c' -> '' move scheduled
I20240503 02:13:25.703125 10317 rebalancer_tool.cc:1074] tablet f14607425da84bb4b162a5789daa844c: '8d7c037d50c241b484ee843b06313a64' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move scheduled
I20240503 02:13:26.115805 10317 rebalancer_tool.cc:1074] tablet 0f44b2b53de84a87a8757c0d00bfed37: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move scheduled
I20240503 02:13:26.437394 10317 rebalancer_tool.cc:1084] tablet 402de43677c44deab8793da1895d4fb2: '2942efc0ecb448d2ba2f6bca20fbb29c' -> '03f99420df704238ab300ad6e58d878c' move ignored: Illegal state: tablet 402de43677c44deab8793da1895d4fb2: replica 03f99420df704238ab300ad6e58d878c is already present
I20240503 02:13:26.437625 10317 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:27.593279 12258 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.0.145.62:36865, user_credentials={real_user=slave}} blocked reactor thread for 52966.5us
W20240503 02:13:30.044939 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:30.045239 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 03f99420df704238ab300ad6e58d878c to server 2942efc0ecb448d2ba2f6bca20fbb29c
W20240503 02:13:30.070212 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:30.070411 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:30.070621 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server 03f99420df704238ab300ad6e58d878c
I20240503 02:13:30.253767 10317 rebalancer_tool.cc:1074] tablet 32e527fa9b2f441cb5668435681b4ff6: '8d7c037d50c241b484ee843b06313a64' -> '03f99420df704238ab300ad6e58d878c' move scheduled
I20240503 02:13:30.447172 10317 rebalancer_tool.cc:1084] tablet 8a8affdde4d8417f95f0585dd01e7d59: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move ignored: Illegal state: tablet 8a8affdde4d8417f95f0585dd01e7d59: replica d826d4dae2aa4e05b2f64c9b17a1126f is already present
I20240503 02:13:30.447367 10317 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:31.884297 12811 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.0.145.62:36865, user_credentials={real_user=slave}} blocked reactor thread for 51282.9us
W20240503 02:13:33.172797 10317 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d
I20240503 02:13:33.252344 10317 rebalancer_tool.cc:1074] tablet 72e2bb75da654e8ab45bba8ac077f28c: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '' move scheduled
I20240503 02:13:33.901870 10317 rebalancer_tool.cc:1136] tablet 32e527fa9b2f441cb5668435681b4ff6: '8d7c037d50c241b484ee843b06313a64' -> '03f99420df704238ab300ad6e58d878c' move completed: OK
I20240503 02:13:34.023092 10317 rebalancer_tool.cc:1136] tablet f14607425da84bb4b162a5789daa844c: '8d7c037d50c241b484ee843b06313a64' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move completed: OK
I20240503 02:13:34.275435 10317 rebalancer_tool.cc:1136] tablet b64f66ace86e42f3ac213ba85bcab584: '2942efc0ecb448d2ba2f6bca20fbb29c' -> '' move completed: OK
I20240503 02:13:34.847959 10317 rebalancer_tool.cc:1123] tablet 834aaba965c8443c862e98a070e36cac: 'd986fcfb926e47baa7d2d5b99f697334' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move is abandoned: Illegal state: 834aaba965c8443c862e98a070e36cac: source replica d986fcfb926e47baa7d2d5b99f697334 does not have REPLACE attribute set
I20240503 02:13:34.952992 10317 rebalancer_tool.cc:1123] tablet 2448e12efbd44a90b1f8bce9519e73e0: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '03f99420df704238ab300ad6e58d878c' move is abandoned: Illegal state: 2448e12efbd44a90b1f8bce9519e73e0: source replica 9cbd8ddb1fc6479eb8cacbfd5b9bf86d does not have REPLACE attribute set
I20240503 02:13:35.026456 10317 rebalancer_tool.cc:1123] tablet aeaee03abc644807b8278cfb78e4f02b: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:35.052938 10317 rebalancer_tool.cc:1136] tablet ecb5c78a22a74728ab40f1da8f90388c: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move completed: OK
I20240503 02:13:35.154064 10317 rebalancer_tool.cc:1136] tablet ceae1c7c45a34b24badbbe23cb9e0176: 'd986fcfb926e47baa7d2d5b99f697334' -> '03f99420df704238ab300ad6e58d878c' move completed: OK
I20240503 02:13:35.178329 10317 rebalancer_tool.cc:1136] tablet cab70b62bdc747a9b653bffde680f37e: '8d7c037d50c241b484ee843b06313a64' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move completed: OK
I20240503 02:13:35.750102 10317 rebalancer_tool.cc:1136] tablet de9c2a3396864696af88be1ca12ee0f1: '2942efc0ecb448d2ba2f6bca20fbb29c' -> '' move completed: OK
I20240503 02:13:35.875619 10317 rebalancer_tool.cc:1084] tablet 80b4e6aa20f64473be8bcbf47baf66e4: '03f99420df704238ab300ad6e58d878c' -> '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' move ignored: Illegal state: tablet 80b4e6aa20f64473be8bcbf47baf66e4: replica 9cbd8ddb1fc6479eb8cacbfd5b9bf86d is already present
I20240503 02:13:35.875787 10317 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): unacceptable health status UNAVAILABLE

I20240503 02:13:37.900166  7826 catalog_manager.cc:5484] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: term changed from 1 to 2, leader changed from 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) to d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1). New cstate: current_term: 2 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:37.944260  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:37.961681  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:38.011252 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:38.128258  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:38.165818  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 80b4e6aa20f64473be8bcbf47baf66e4: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:13:38.277923  9995 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:38.278595 13704 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:38.285281  8100 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:38.285827 13704 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:38.289791  8230 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:38.290369 13704 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:13:38.395089  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:38.588173 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:13:38.806090  7969 consensus_queue.cc:237] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:13:38.807783  9995 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:38.807991  8230 raft_consensus.cc:1261] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Refusing update from remote peer d986fcfb926e47baa7d2d5b99f697334: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:38.808385 14081 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:38.808717 14082 consensus_queue.cc:1035] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:38.811095 13988 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:13:38.811859  8230 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:13:38.813900  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 45fe6ec58eed40f1898d4a2f5727cfa1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:38.814445  9995 raft_consensus.cc:2932] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:13:38.816054  7826 catalog_manager.cc:5484] T 45fe6ec58eed40f1898d4a2f5727cfa1 P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 8 to 10, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New cstate: current_term: 2 leader_uuid: "d986fcfb926e47baa7d2d5b99f697334" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:38.822158  8081 tablet_service.cc:1514] Processing DeleteTablet for tablet 45fe6ec58eed40f1898d4a2f5727cfa1 with delete_type TABLET_DATA_TOMBSTONED (TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:54670
I20240503 02:13:38.822875 14086 tablet_replica.cc:330] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: stopping tablet replica
I20240503 02:13:38.823132 14086 raft_consensus.cc:2229] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:38.823343 14086 raft_consensus.cc:2258] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:38.824194 14086 ts_tablet_manager.cc:1875] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:38.828261 14086 ts_tablet_manager.cc:1888] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:13:38.828368 14086 log.cc:1196] T 45fe6ec58eed40f1898d4a2f5727cfa1 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/45fe6ec58eed40f1898d4a2f5727cfa1
I20240503 02:13:38.828954  7816 catalog_manager.cc:4830] TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): tablet 45fe6ec58eed40f1898d4a2f5727cfa1 (table rebalance_test_table_9 [id=5f6139f12f9b4ab18ce35a8831b39150]) successfully deleted
W20240503 02:13:38.885761  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:39.011150  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:39.108021 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:39.187520  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:39.200953  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 80b4e6aa20f64473be8bcbf47baf66e4: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:39.441435  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:39.804086 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:39.934033  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:40.019737 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:40.251704  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:13:40.338541 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:13:40.342137  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:13:40.343269  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:40.369207  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:13:40.370153  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:40.389771  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:40.397874  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:40.398689 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:13:40.399546 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:40.803251  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:40.830994  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:40.965727 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:41.218039  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:41.294445  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 80b4e6aa20f64473be8bcbf47baf66e4: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:41.594415 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:41.599615  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:41.602870  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:41.845777  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:42.159672 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:42.220772  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:42.594206 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:42.783803  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:42.784276 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:42.806341  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:42.810393  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:42.882009  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:43.264950  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:43.401618  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:43.674264 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:43.684376  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:44.109928  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:44.149868  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:44.211764  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:44.404631 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:44.517136 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:44.804522  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:44.953548 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:44.976563  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:45.072117  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:45.302409  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:45.307987 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:45.414413  7817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240503 02:13:45.415544  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 80b4e6aa20f64473be8bcbf47baf66e4: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:45.484100  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:13:45.485177  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:45.488657  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:13:45.546278  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:13:45.594718 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:13:45.599941 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:13:45.677374  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:46.047276  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:46.080896 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:46.157406  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:46.442718  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:46.626886  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:46.778007  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:46.955392 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:47.020536 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:47.167496  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:47.525772 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:47.655381  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:47.667605  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:47.691432  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:47.840153 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:48.058526  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:48.191154  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:48.568934  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:48.649145  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:48.723361 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:48.837301  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:48.960937  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:49.001499  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:49.301878 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:49.376467 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:49.490036  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:49.865325  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:50.019567 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:50.143615  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:50.248258  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:50.395877  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:50.432031 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:50.509531  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:13:50.510318  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:50.555047  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:13:50.669566 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:13:50.810242  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:13:50.896365 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:13:50.913278  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:51.090344  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:51.099876 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:51.273766  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:51.385907  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:51.523309  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:51.783463 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:51.812167  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:52.022264 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:52.227811  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:52.392385 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:52.403427  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:52.816646  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:52.857228  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:52.907954  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:52.950538 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:53.308689  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:53.485388  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:53.620630  7817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:13:53.621701  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 80b4e6aa20f64473be8bcbf47baf66e4: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:53.627722  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:53.839272 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:53.847455  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:53.899978  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:54.112937 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:54.467859  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:54.567778 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:54.791042 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:54.947201  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:54.978422  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:55.076784 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:55.488792  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:55.550976  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:13:55.551863  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:55.597699  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:55.700400 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:13:55.736397  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:55.972872  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:13:56.010311 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:13:56.079644  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:56.127676  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:13:56.128659  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:56.157943 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:56.344051  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:56.469657 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:56.571244  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:57.030490 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:57.059243  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:57.155414 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:57.530957  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:57.592486  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:57.682837 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:57.945705  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:57.984854  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:58.077179  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:58.342689  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:58.422470  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:58.676144 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:58.676880  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:58.880501 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:59.048688  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:59.078229  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:59.535322  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:13:59.604357 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:59.644011 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:59.755369  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:59.978945  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:00.031453 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:14:00.291245  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:14:00.530664  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:00.551745  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:14:00.553361  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:14:00.554597  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:14:00.778391 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:14:00.913894  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:01.005074 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:14:01.044066 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:14:01.283617  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:14:01.284505  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:14:01.287348  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:14:01.419744  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:01.490135  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:14:01.650014 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:14:01.884568  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:02.168220 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:02.243595 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:02.278914  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:02.333428  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:02.715858  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:02.747092 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:02.821728  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:02.896160  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:03.080363  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:03.387560  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:03.429425 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:03.899544  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:04.029819  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:04.176817  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:04.233351 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:04.443305  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:04.795302 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:04.797129 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:04.908757  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:05.097707  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:05.191023  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:05.268002 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:05.357434  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:05.402479  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:05.560683  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:14:05.563338  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:05.784472  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:05.905038 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:14:05.905985 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:06.183717 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:14:06.310521  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:06.480513  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:14:06.481623  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:06.600390  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:14:06.639616  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:06.661919 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:06.829111  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:06.934408 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:07.240226  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:07.348125 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:07.540190  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:07.757802  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:07.790012 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:07.806486  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:07.843994  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:08.080924  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:08.143236  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:08.640704  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:08.666918 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:08.855670  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:08.968158 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:08.983738  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:09.136693 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:09.243949  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:09.614114 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:09.792212  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:10.024482  7817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:14:10.026034  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 80b4e6aa20f64473be8bcbf47baf66e4: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:10.086076  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:10.253032  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:10.406422  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:10.451421  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:10.659215 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:10.677327  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:14:10.679672  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:10.753151  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:11.159727 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:14:11.160941 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:11.186792  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:11.294346  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:11.448462  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:11.475497 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:14:11.476581 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:11.611176  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:14:11.739395 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:11.880178  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:12.143935  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:14:12.188174 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:12.401907  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:12.673308  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:12.674361  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:12.862704  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:12.931377  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:13.001528  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:13.483095 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:13.538272  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:13.647089  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:13.772286 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:13.817427 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:13.924983  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:14.220250 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:14.285320  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:14.393829  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:14.729225 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:14.785679  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:14.939260  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:15.178627  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:15.206287  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:15.529595  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:15.584372  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:15.714356  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:14:15.715306  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:16.044083 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:16.132930  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:16.185599 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:14:16.202109 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:16.211747  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:16.277002 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:16.511221  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:16.628505 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:16.653443 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:16.660671  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:14:16.661681  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:17.004137  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:17.200098 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:17.389804  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:14:17.392156  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:17.515588  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:17.743311  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:17.852494  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:18.109489  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:18.283246 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:18.365027  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:18.518014 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:18.610435  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:18.824599  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:18.919268 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:19.088814  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:19.191277 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:19.251399  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:19.373733 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:19.647290  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:20.035554  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:20.036571  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:20.561246  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:20.564615  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:20.598326  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:20.629642  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:20.828161 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:20.975678  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:14:21.052381  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:21.163944 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:21.330555 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:14:21.362715  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:21.435523 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:21.503476  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:21.558236  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:21.731492 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:14:21.798709 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:21.852525  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:22.075870  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:22.137609 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:22.522259  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:22.575390  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:14:22.592090  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:22.924190  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:22.965579  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:23.029831 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:23.163527  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:23.208045  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:23.409384  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:23.600523 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:23.690043  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:23.717120 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:23.890341  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:23.928810  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:24.198947 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:24.381433  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:24.649971 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:24.918900  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:25.085558  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:25.236485  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:25.384552  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:25.446499 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:25.684964  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:25.707551  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:25.987285  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:14:25.988112  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:26.158140 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:26.175896 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:26.214936  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:26.375070 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:14:26.460562  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:26.574766  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:26.848750 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:14:26.894694 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:27.079277  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:27.169765  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:27.335533 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:27.508777  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:27.733713  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:27.859083 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:27.971549  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:27.984561  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:14:28.136919  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:28.475421  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:28.526866  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:28.638023  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:28.655499 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:28.768353 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:28.805828  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:28.910488  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:29.080889 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:29.203889  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:29.697641  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:29.969240  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:30.061094  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:30.075138 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:30.093619  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:30.442699 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:30.657835  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:30.831447  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:30.849076  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:30.984932 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:31.049810  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:31.102777  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:14:31.241760  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:31.322707 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:31.432955  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:31.445798 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:14:31.605090 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:31.811640  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:31.937780 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:14:32.249152  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:14:32.397776  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:32.414674  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:32.588181 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:32.656327  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:32.830302 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:32.933722  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:32.995103  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:14:33.251310  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:33.375478  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:33.400758 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:33.502557  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:33.582288  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:33.772531  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:33.994804 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:34.101756  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:34.499090 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:34.546736  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:34.983400  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:35.046324  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:35.201732  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:35.257668 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:35.281956 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:35.459115  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:35.833527  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:35.922634  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:35.982849  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:36.021256 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:36.246842  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:36.295007  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:14:36.494580 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240503 02:14:36.496259 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:36.524396  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:36.591463  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:37.005146  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:37.071233 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:37.113364 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:37.315920  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:14:37.436466  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:37.617247  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:37.643930 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:37.711973  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:37.736043 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:38.059429  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:38.153666  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:38.191573  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:14:38.419134  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:38.439493 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:38.497520  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:38.608045  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:38.937824  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:38.976022 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:38.988265  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:39.395936  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:39.551286 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:40.003264  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:40.152624 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:40.165913 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:40.231608  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:40.372160  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:40.509321  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:40.538746  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:40.954048  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:41.078250  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:41.197679 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:41.203436  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:41.279212  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:41.423861  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:14:41.424820  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:41.623256 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:14:41.662379 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:41.931098 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:42.036192  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:42.220834 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:42.367183  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:42.403592 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:42.638942  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:42.775904 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:42.810441  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:42.827842  7817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:14:42.828951  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 80b4e6aa20f64473be8bcbf47baf66e4: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:42.843472  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:43.149894  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:43.284155  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:43.353206  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:14:43.490325  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:43.602015  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:43.743029 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:43.763758  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:43.882444  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:44.194296  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:44.324095 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:44.572453  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:44.707973 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:44.967808  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:44.999655 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:45.340669  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:45.442708 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:45.580684  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:45.614408  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:45.656194  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:45.965463  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:46.056640  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:46.189472 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:46.373567  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:46.445820  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:14:46.446686  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:46.636519  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:46.730651 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:14:46.888329  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:47.085223 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:47.407367  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:47.442557 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:14:47.477715 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:47.510226 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:47.699819  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:47.706224  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:14:47.918200  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:47.967830 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:48.205371  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:48.361361  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:48.373701  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:48.411753  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:48.634033  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:14:48.711249 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:48.866086  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:48.893355  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:49.292198  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:49.415143  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:49.741535 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:49.750844  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:50.039323 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:50.191561  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:50.337872  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:50.389186 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:50.675796  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:50.717761 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:50.748634  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:50.991472 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:51.059825  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:51.129622  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:51.135947  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:14:51.193373  7819 maintenance_manager.cc:419] P 5680e1b3edfe402b831ab3889e06b093: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033370
W20240503 02:14:51.229751  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:14:51.240520  7818 maintenance_manager.cc:643] P 5680e1b3edfe402b831ab3889e06b093: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.046s	user 0.036s	sys 0.008s Metrics: {"bytes_written":20641,"cfile_init":1,"dirs.queue_time_us":344,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":1313,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":874,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":43,"thread_start_us":411,"threads_started":2,"wal-append.queue_time_us":287}
I20240503 02:14:51.241747  7819 maintenance_manager.cc:419] P 5680e1b3edfe402b831ab3889e06b093: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 11218 bytes on disk
I20240503 02:14:51.242488  7818 maintenance_manager.cc:643] P 5680e1b3edfe402b831ab3889e06b093: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240503 02:14:51.532405  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:14:51.826732  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:51.885773 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:14:51.984690  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:52.177182 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:52.243036  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:52.243750 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:52.592597  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:52.681610 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:52.683974 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:52.748816  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:52.874038  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:14:53.062999 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:53.105620  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:53.235567  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:53.359138  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:53.376303 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:53.671881  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:53.789413  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:53.817682  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:54.070266  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:14:54.328092  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:54.471879 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:54.717554  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:54.809623  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:54.819869 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:54.989387  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:55.144855 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:55.229918  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:55.611418  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:55.654361 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:55.677258  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:56.020604  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:56.161330  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:56.165611  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:56.188918 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:56.221777  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:56.457715  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:56.745172  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:14:56.926187 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:14:57.033500  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:57.179620 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:57.400403  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:57.447780 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:57.454710  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:57.459203  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:57.585662 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:57.696102 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:14:57.854326  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:57.912833  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:14:58.238541 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:58.320963  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:58.383618  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:58.434645  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:58.500577  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:58.832749  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:58.892123 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:58.911706  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:59.296881  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:59.365394  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:14:59.453310  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling FlushMRSOp(2d69c71d9dfd47a1a2429e7caf5e33ac): perf score=0.033341
I20240503 02:14:59.494148  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling FlushMRSOp(80b4e6aa20f64473be8bcbf47baf66e4): perf score=0.033362
I20240503 02:14:59.552791  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: FlushMRSOp(2d69c71d9dfd47a1a2429e7caf5e33ac) complete. Timing: real 0.099s	user 0.085s	sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":320,"dirs.queue_time_us":398,"dirs.run_cpu_time_us":270,"dirs.run_wall_time_us":1245,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":890,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":28269,"threads_started":3,"wal-append.queue_time_us":27882}
I20240503 02:14:59.555410  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(2d69c71d9dfd47a1a2429e7caf5e33ac): 4583 bytes on disk
I20240503 02:14:59.556425  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(2d69c71d9dfd47a1a2429e7caf5e33ac) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240503 02:14:59.596297  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: FlushMRSOp(80b4e6aa20f64473be8bcbf47baf66e4) complete. Timing: real 0.101s	user 0.055s	sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":2206,"dirs.queue_time_us":384,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":972,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":914,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":2619,"threads_started":3,"wal-append.queue_time_us":372}
I20240503 02:14:59.598593  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling UndoDeltaBlockGCOp(80b4e6aa20f64473be8bcbf47baf66e4): 4923 bytes on disk
I20240503 02:14:59.599630  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: UndoDeltaBlockGCOp(80b4e6aa20f64473be8bcbf47baf66e4) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240503 02:14:59.635478  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling FlushMRSOp(f14607425da84bb4b162a5789daa844c): perf score=0.033392
W20240503 02:14:59.661518 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:59.702708 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:59.705207  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:14:59.829210  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:59.853943 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:14:59.911305  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: FlushMRSOp(f14607425da84bb4b162a5789daa844c) complete. Timing: real 0.274s	user 0.095s	sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":31729,"dirs.queue_time_us":1805,"dirs.run_cpu_time_us":281,"dirs.run_wall_time_us":7515,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1069,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":44359,"threads_started":3,"wal-append.queue_time_us":12294}
I20240503 02:14:59.917400  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling UndoDeltaBlockGCOp(f14607425da84bb4b162a5789daa844c): 4838 bytes on disk
I20240503 02:14:59.919667  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: UndoDeltaBlockGCOp(f14607425da84bb4b162a5789daa844c) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
W20240503 02:14:59.967371  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:15:00.215991  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:00.650215  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:00.674866  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling FlushMRSOp(32e527fa9b2f441cb5668435681b4ff6): perf score=0.033348
I20240503 02:15:00.683502  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(3dbfc7f12fc142f98b1d1c026fd49f5a): perf score=0.033359
I20240503 02:15:00.712188  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: FlushMRSOp(32e527fa9b2f441cb5668435681b4ff6) complete. Timing: real 0.037s	user 0.031s	sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":387,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1716,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":574,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":655,"threads_started":2,"wal-append.queue_time_us":496}
I20240503 02:15:00.713730  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling UndoDeltaBlockGCOp(32e527fa9b2f441cb5668435681b4ff6): 4923 bytes on disk
I20240503 02:15:00.714527  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: UndoDeltaBlockGCOp(32e527fa9b2f441cb5668435681b4ff6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240503 02:15:00.748842  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(3dbfc7f12fc142f98b1d1c026fd49f5a) complete. Timing: real 0.064s	user 0.043s	sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":447,"dirs.queue_time_us":432,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":1035,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":788,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":937,"threads_started":3,"wal-append.queue_time_us":435}
I20240503 02:15:00.751191  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(8a8affdde4d8417f95f0585dd01e7d59): perf score=0.033376
I20240503 02:15:00.805317  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(8a8affdde4d8417f95f0585dd01e7d59) complete. Timing: real 0.054s	user 0.035s	sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":1575,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":736,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":341,"threads_started":1,"wal-append.queue_time_us":453}
I20240503 02:15:00.806947  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(8a8affdde4d8417f95f0585dd01e7d59): 4838 bytes on disk
I20240503 02:15:00.807866  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(8a8affdde4d8417f95f0585dd01e7d59) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240503 02:15:00.809137  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(3dbfc7f12fc142f98b1d1c026fd49f5a): 4583 bytes on disk
I20240503 02:15:00.809888  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(3dbfc7f12fc142f98b1d1c026fd49f5a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 02:15:00.813668  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling FlushMRSOp(8a8affdde4d8417f95f0585dd01e7d59): perf score=0.033382
W20240503 02:15:00.845539  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:15:00.854717 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:15:00.856184  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling FlushMRSOp(3dbfc7f12fc142f98b1d1c026fd49f5a): perf score=0.033401
W20240503 02:15:00.929718  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:15:00.937637  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: FlushMRSOp(8a8affdde4d8417f95f0585dd01e7d59) complete. Timing: real 0.124s	user 0.071s	sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":15180,"dirs.run_cpu_time_us":314,"dirs.run_wall_time_us":1713,"drs_written":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1842,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16478,"threads_started":2,"wal-append.queue_time_us":7841}
I20240503 02:15:00.939842  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(8a8affdde4d8417f95f0585dd01e7d59): 4838 bytes on disk
I20240503 02:15:00.940496  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(8a8affdde4d8417f95f0585dd01e7d59) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
W20240503 02:15:00.942835  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:15:00.968345  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: FlushMRSOp(3dbfc7f12fc142f98b1d1c026fd49f5a) complete. Timing: real 0.112s	user 0.047s	sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":17729,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":1524,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":954,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":26376,"threads_started":2,"wal-append.queue_time_us":5468}
I20240503 02:15:00.970062  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling UndoDeltaBlockGCOp(3dbfc7f12fc142f98b1d1c026fd49f5a): 4583 bytes on disk
I20240503 02:15:00.970824  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: UndoDeltaBlockGCOp(3dbfc7f12fc142f98b1d1c026fd49f5a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240503 02:15:01.122505  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:15:01.164674  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:01.554044 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:15:01.652319  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:01.696273  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling FlushMRSOp(cab70b62bdc747a9b653bffde680f37e): perf score=0.033346
I20240503 02:15:01.719707  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling FlushMRSOp(5ba1f505010b403c91d3870cee19952a): perf score=0.033351
I20240503 02:15:01.755206  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: FlushMRSOp(cab70b62bdc747a9b653bffde680f37e) complete. Timing: real 0.059s	user 0.045s	sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":621,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":6982,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":724,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":15233,"threads_started":2,"wal-append.queue_time_us":14844}
I20240503 02:15:01.756762  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(cab70b62bdc747a9b653bffde680f37e): 4923 bytes on disk
I20240503 02:15:01.757450  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(cab70b62bdc747a9b653bffde680f37e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240503 02:15:01.758541  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling FlushMRSOp(5ba1f505010b403c91d3870cee19952a): perf score=0.033358
I20240503 02:15:01.791160  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: FlushMRSOp(5ba1f505010b403c91d3870cee19952a) complete. Timing: real 0.071s	user 0.057s	sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":328,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1449,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":893,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":451,"threads_started":2,"wal-append.queue_time_us":334}
I20240503 02:15:01.792315  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling UndoDeltaBlockGCOp(5ba1f505010b403c91d3870cee19952a): 4583 bytes on disk
I20240503 02:15:01.792876  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: UndoDeltaBlockGCOp(5ba1f505010b403c91d3870cee19952a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240503 02:15:01.803970  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: FlushMRSOp(5ba1f505010b403c91d3870cee19952a) complete. Timing: real 0.045s	user 0.019s	sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":1701,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":793,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":228,"threads_started":1,"wal-append.queue_time_us":350}
I20240503 02:15:01.805051  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(5ba1f505010b403c91d3870cee19952a): 4583 bytes on disk
I20240503 02:15:01.805549  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(5ba1f505010b403c91d3870cee19952a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240503 02:15:01.814988  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(0f44b2b53de84a87a8757c0d00bfed37): perf score=0.033373
I20240503 02:15:01.848105  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(0f44b2b53de84a87a8757c0d00bfed37) complete. Timing: real 0.033s	user 0.018s	sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":447,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":1477,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":658,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":565,"threads_started":2,"wal-append.queue_time_us":351}
I20240503 02:15:01.849522  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(0f44b2b53de84a87a8757c0d00bfed37): 4836 bytes on disk
I20240503 02:15:01.850030  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(0f44b2b53de84a87a8757c0d00bfed37) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240503 02:15:01.882300  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:15:01.915844 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:15:01.935328 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:15:02.258762  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:02.300354 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:02.416263 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:02.566906  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:15:02.585525  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:02.658154  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:02.728734  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling FlushMRSOp(402de43677c44deab8793da1895d4fb2): perf score=0.033336
I20240503 02:15:02.765414  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: FlushMRSOp(402de43677c44deab8793da1895d4fb2) complete. Timing: real 0.036s	user 0.030s	sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":438,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1704,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":698,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":494,"threads_started":2,"wal-append.queue_time_us":375}
I20240503 02:15:02.766957  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling UndoDeltaBlockGCOp(402de43677c44deab8793da1895d4fb2): 4838 bytes on disk
I20240503 02:15:02.767465  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: UndoDeltaBlockGCOp(402de43677c44deab8793da1895d4fb2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240503 02:15:02.796602  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling FlushMRSOp(6043da7d2e584af885ff6a5bf8f06912): perf score=0.033355
I20240503 02:15:02.810120  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling FlushMRSOp(6043da7d2e584af885ff6a5bf8f06912): perf score=0.033359
I20240503 02:15:02.828421  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: FlushMRSOp(6043da7d2e584af885ff6a5bf8f06912) complete. Timing: real 0.032s	user 0.022s	sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":411,"dirs.run_cpu_time_us":281,"dirs.run_wall_time_us":1533,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":605,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":8576,"thread_start_us":617,"threads_started":2,"wal-append.queue_time_us":511}
I20240503 02:15:02.829510  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling UndoDeltaBlockGCOp(6043da7d2e584af885ff6a5bf8f06912): 4583 bytes on disk
I20240503 02:15:02.830180  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: UndoDeltaBlockGCOp(6043da7d2e584af885ff6a5bf8f06912) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240503 02:15:02.839730 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 02:15:02.855283  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(007b59de142c4b95966494ba097b1a09): perf score=0.033378
I20240503 02:15:02.861944  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: FlushMRSOp(6043da7d2e584af885ff6a5bf8f06912) complete. Timing: real 0.052s	user 0.046s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":939,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1542,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1659,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1039,"threads_started":2,"wal-append.queue_time_us":344}
I20240503 02:15:02.863408  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(6043da7d2e584af885ff6a5bf8f06912): 4583 bytes on disk
I20240503 02:15:02.863960  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(6043da7d2e584af885ff6a5bf8f06912) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 02:15:02.905607  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(007b59de142c4b95966494ba097b1a09) complete. Timing: real 0.050s	user 0.043s	sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":327,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1735,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":769,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":446,"threads_started":2,"wal-append.queue_time_us":344}
I20240503 02:15:02.906914  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(007b59de142c4b95966494ba097b1a09): 4921 bytes on disk
I20240503 02:15:02.907430  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(007b59de142c4b95966494ba097b1a09) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240503 02:15:02.908531  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(402de43677c44deab8793da1895d4fb2): perf score=0.033389
I20240503 02:15:02.943018  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(402de43677c44deab8793da1895d4fb2) complete. Timing: real 0.034s	user 0.022s	sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":68,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":1480,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":612,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":220,"threads_started":1,"wal-append.queue_time_us":332}
I20240503 02:15:02.944650  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(402de43677c44deab8793da1895d4fb2): 4838 bytes on disk
I20240503 02:15:02.945271  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(402de43677c44deab8793da1895d4fb2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240503 02:15:02.980226  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:15:03.129323  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:03.272010 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:03.324558  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:03.454289  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:03.528240  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:03.562108  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:03.688704  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:15:03.950563  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(ceae1c7c45a34b24badbbe23cb9e0176): perf score=0.033363
W20240503 02:15:03.963474  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:03.989648  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(ceae1c7c45a34b24badbbe23cb9e0176) complete. Timing: real 0.039s	user 0.017s	sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":346,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1466,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":637,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":455,"threads_started":2,"wal-append.queue_time_us":374}
I20240503 02:15:03.991022  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(ceae1c7c45a34b24badbbe23cb9e0176): 4923 bytes on disk
I20240503 02:15:03.991607  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(ceae1c7c45a34b24badbbe23cb9e0176) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 02:15:04.022557  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling FlushMRSOp(d3d4b912095244de816d3089a31b8649): perf score=0.033376
I20240503 02:15:04.085300  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling FlushMRSOp(ceae1c7c45a34b24badbbe23cb9e0176): perf score=0.033400
I20240503 02:15:04.089713  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: FlushMRSOp(d3d4b912095244de816d3089a31b8649) complete. Timing: real 0.067s	user 0.053s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":5687,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":1683,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":811,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":7552,"thread_start_us":5799,"threads_started":2,"wal-append.queue_time_us":329}
I20240503 02:15:04.090960  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling UndoDeltaBlockGCOp(d3d4b912095244de816d3089a31b8649): 4838 bytes on disk
I20240503 02:15:04.091598  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: UndoDeltaBlockGCOp(d3d4b912095244de816d3089a31b8649) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
W20240503 02:15:04.094430 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:15:04.120357  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling FlushMRSOp(5ea56405515a40f785726bd58ec3d26a): perf score=0.033401
I20240503 02:15:04.123000  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: FlushMRSOp(ceae1c7c45a34b24badbbe23cb9e0176) complete. Timing: real 0.037s	user 0.024s	sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":866,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1633,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":761,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":935,"threads_started":2,"wal-append.queue_time_us":322}
I20240503 02:15:04.124162  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling UndoDeltaBlockGCOp(ceae1c7c45a34b24badbbe23cb9e0176): 4923 bytes on disk
I20240503 02:15:04.124698  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: UndoDeltaBlockGCOp(ceae1c7c45a34b24badbbe23cb9e0176) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240503 02:15:04.178052  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: FlushMRSOp(5ea56405515a40f785726bd58ec3d26a) complete. Timing: real 0.057s	user 0.022s	sys 0.028s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":11101,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1460,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":778,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11197,"threads_started":2,"wal-append.queue_time_us":349}
I20240503 02:15:04.179600  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(5ea56405515a40f785726bd58ec3d26a): 4583 bytes on disk
I20240503 02:15:04.180217  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(5ea56405515a40f785726bd58ec3d26a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240503 02:15:04.269048 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:04.426061  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:04.455504  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:15:04.728298 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:04.884620 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:04.906811  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:04.994994  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:05.095742  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling FlushMRSOp(7e966caa651e4e42890be5d79f851a1f): perf score=0.033357
I20240503 02:15:05.126264  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: FlushMRSOp(7e966caa651e4e42890be5d79f851a1f) complete. Timing: real 0.030s	user 0.024s	sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":427,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":1792,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":587,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":512,"threads_started":2,"wal-append.queue_time_us":346}
I20240503 02:15:05.127704  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling UndoDeltaBlockGCOp(7e966caa651e4e42890be5d79f851a1f): 4919 bytes on disk
I20240503 02:15:05.128417  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: UndoDeltaBlockGCOp(7e966caa651e4e42890be5d79f851a1f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240503 02:15:05.128901  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling FlushMRSOp(7e966caa651e4e42890be5d79f851a1f): perf score=0.033371
I20240503 02:15:05.129885  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling FlushMRSOp(2448e12efbd44a90b1f8bce9519e73e0): perf score=0.033362
I20240503 02:15:05.185467  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling FlushMRSOp(4f2cedba9f9a4e709623fceeffcfd095): perf score=0.033383
W20240503 02:15:05.215819  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:15:05.255877  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: FlushMRSOp(2448e12efbd44a90b1f8bce9519e73e0) complete. Timing: real 0.126s	user 0.056s	sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1953,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1192,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":16762,"threads_started":1,"wal-append.queue_time_us":16907}
I20240503 02:15:05.259344  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling UndoDeltaBlockGCOp(2448e12efbd44a90b1f8bce9519e73e0): 4581 bytes on disk
I20240503 02:15:05.261798  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: UndoDeltaBlockGCOp(2448e12efbd44a90b1f8bce9519e73e0) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":146,"lbm_reads_lt_1ms":4}
I20240503 02:15:05.292358  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: FlushMRSOp(4f2cedba9f9a4e709623fceeffcfd095) complete. Timing: real 0.107s	user 0.051s	sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":25927,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1414,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1643,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":29232,"threads_started":2,"wal-append.queue_time_us":6782}
I20240503 02:15:05.293896  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(4f2cedba9f9a4e709623fceeffcfd095): 4836 bytes on disk
I20240503 02:15:05.294685  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(4f2cedba9f9a4e709623fceeffcfd095) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240503 02:15:05.301391  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: FlushMRSOp(7e966caa651e4e42890be5d79f851a1f) complete. Timing: real 0.169s	user 0.069s	sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3282,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1461,"drs_written":1,"lbm_read_time_us":139,"lbm_reads_lt_1ms":4,"lbm_write_time_us":802,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5136,"threads_started":2,"wal-append.queue_time_us":1017}
I20240503 02:15:05.302886  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling UndoDeltaBlockGCOp(7e966caa651e4e42890be5d79f851a1f): 4919 bytes on disk
I20240503 02:15:05.303563  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: UndoDeltaBlockGCOp(7e966caa651e4e42890be5d79f851a1f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
I20240503 02:15:05.304615  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling FlushMRSOp(2448e12efbd44a90b1f8bce9519e73e0): perf score=0.033415
I20240503 02:15:05.356578  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: FlushMRSOp(2448e12efbd44a90b1f8bce9519e73e0) complete. Timing: real 0.052s	user 0.039s	sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":115,"dirs.run_cpu_time_us":261,"dirs.run_wall_time_us":1580,"drs_written":1,"lbm_read_time_us":207,"lbm_reads_lt_1ms":4,"lbm_write_time_us":989,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3080,"threads_started":1,"wal-append.queue_time_us":3697}
I20240503 02:15:05.357793  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling UndoDeltaBlockGCOp(2448e12efbd44a90b1f8bce9519e73e0): 4581 bytes on disk
I20240503 02:15:05.358424  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: UndoDeltaBlockGCOp(2448e12efbd44a90b1f8bce9519e73e0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240503 02:15:05.573575  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:05.634430 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:05.758081  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:05.891999  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:05.967705  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:06.176259  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:15:06.250592  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(7e4acc8b5ceb4cd5b0f3708ff4f19298): perf score=0.033365
I20240503 02:15:06.266948  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling FlushMRSOp(7e4acc8b5ceb4cd5b0f3708ff4f19298): perf score=0.033363
I20240503 02:15:06.288905  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(7e4acc8b5ceb4cd5b0f3708ff4f19298) complete. Timing: real 0.038s	user 0.031s	sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":415,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1677,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":885,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":599,"threads_started":2,"wal-append.queue_time_us":393}
I20240503 02:15:06.291213  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(7e4acc8b5ceb4cd5b0f3708ff4f19298): 4838 bytes on disk
I20240503 02:15:06.292220  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(7e4acc8b5ceb4cd5b0f3708ff4f19298) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240503 02:15:06.299651  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling FlushMRSOp(f231fe5dfd9e4c87b90bad75818a7060): perf score=0.033380
I20240503 02:15:06.354516  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: FlushMRSOp(7e4acc8b5ceb4cd5b0f3708ff4f19298) complete. Timing: real 0.087s	user 0.051s	sys 0.025s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":441,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":2813,"drs_written":1,"lbm_read_time_us":134,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1127,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2665,"threads_started":2,"wal-append.queue_time_us":2191}
I20240503 02:15:06.356995  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling UndoDeltaBlockGCOp(7e4acc8b5ceb4cd5b0f3708ff4f19298): 4838 bytes on disk
I20240503 02:15:06.358026  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: UndoDeltaBlockGCOp(7e4acc8b5ceb4cd5b0f3708ff4f19298) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":157,"lbm_reads_lt_1ms":4}
I20240503 02:15:06.364020  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling FlushMRSOp(f231fe5dfd9e4c87b90bad75818a7060): perf score=0.033393
W20240503 02:15:06.392839  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:15:06.435918  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: FlushMRSOp(f231fe5dfd9e4c87b90bad75818a7060) complete. Timing: real 0.136s	user 0.065s	sys 0.029s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6802,"dirs.run_cpu_time_us":278,"dirs.run_wall_time_us":2227,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1000,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":12318,"threads_started":2,"wal-append.queue_time_us":5664}
I20240503 02:15:06.437552  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling FlushMRSOp(c45c5a14b85447c78fe2a27be4392c88): perf score=0.033412
I20240503 02:15:06.461910  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: FlushMRSOp(f231fe5dfd9e4c87b90bad75818a7060) complete. Timing: real 0.097s	user 0.066s	sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":537,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":6645,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":989,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":9539,"threads_started":2,"wal-append.queue_time_us":8745}
I20240503 02:15:06.463455  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling UndoDeltaBlockGCOp(f231fe5dfd9e4c87b90bad75818a7060): 4583 bytes on disk
I20240503 02:15:06.464960  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: UndoDeltaBlockGCOp(f231fe5dfd9e4c87b90bad75818a7060) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240503 02:15:06.486631  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: FlushMRSOp(c45c5a14b85447c78fe2a27be4392c88) complete. Timing: real 0.049s	user 0.019s	sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":440,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":1524,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":692,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":223,"threads_started":1,"wal-append.queue_time_us":343}
W20240503 02:15:06.487362  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:06.487811  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(c45c5a14b85447c78fe2a27be4392c88): 4923 bytes on disk
I20240503 02:15:06.488327  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(c45c5a14b85447c78fe2a27be4392c88) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240503 02:15:06.489326  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(f231fe5dfd9e4c87b90bad75818a7060): 4583 bytes on disk
I20240503 02:15:06.490010  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(f231fe5dfd9e4c87b90bad75818a7060) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240503 02:15:06.534258 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:06.978313  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:15:06.979138  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:07.072662 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:15:07.119050 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:15:07.243811  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling FlushMRSOp(ecb5c78a22a74728ab40f1da8f90388c): perf score=0.033348
I20240503 02:15:07.276937  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: FlushMRSOp(ecb5c78a22a74728ab40f1da8f90388c) complete. Timing: real 0.033s	user 0.030s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":331,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1427,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":618,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":488,"threads_started":2,"wal-append.queue_time_us":385}
I20240503 02:15:07.278113  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(ecb5c78a22a74728ab40f1da8f90388c): 4923 bytes on disk
I20240503 02:15:07.278674  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(ecb5c78a22a74728ab40f1da8f90388c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 02:15:07.279605  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling FlushMRSOp(c23b06764ce54acb8388de94aea4cd4b): perf score=0.033351
I20240503 02:15:07.297482  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(de9c2a3396864696af88be1ca12ee0f1): perf score=0.033359
I20240503 02:15:07.332022  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: FlushMRSOp(c23b06764ce54acb8388de94aea4cd4b) complete. Timing: real 0.052s	user 0.043s	sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1492,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":820,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":2304,"thread_start_us":326,"threads_started":1,"wal-append.queue_time_us":453}
I20240503 02:15:07.333755  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(c23b06764ce54acb8388de94aea4cd4b): 4581 bytes on disk
I20240503 02:15:07.334345  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(c23b06764ce54acb8388de94aea4cd4b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 02:15:07.338699  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(de9c2a3396864696af88be1ca12ee0f1) complete. Timing: real 0.041s	user 0.031s	sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":370,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1498,"drs_written":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4,"lbm_write_time_us":692,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":488,"threads_started":2,"wal-append.queue_time_us":338}
I20240503 02:15:07.339805  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(de9c2a3396864696af88be1ca12ee0f1): 4838 bytes on disk
I20240503 02:15:07.340332  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(de9c2a3396864696af88be1ca12ee0f1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240503 02:15:07.357656  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:07.363286  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling FlushMRSOp(de9c2a3396864696af88be1ca12ee0f1): perf score=0.033381
W20240503 02:15:07.382166 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:15:07.395706  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: FlushMRSOp(de9c2a3396864696af88be1ca12ee0f1) complete. Timing: real 0.032s	user 0.024s	sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":316,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1505,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":771,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":431,"threads_started":2,"wal-append.queue_time_us":334}
I20240503 02:15:07.396785  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling FlushMRSOp(c23b06764ce54acb8388de94aea4cd4b): perf score=0.033384
W20240503 02:15:07.422566 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:15:07.434202  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: FlushMRSOp(c23b06764ce54acb8388de94aea4cd4b) complete. Timing: real 0.037s	user 0.025s	sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1584,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":701,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":257,"threads_started":1,"wal-append.queue_time_us":394}
I20240503 02:15:07.436164  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling UndoDeltaBlockGCOp(de9c2a3396864696af88be1ca12ee0f1): 4838 bytes on disk
I20240503 02:15:07.437487  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: UndoDeltaBlockGCOp(de9c2a3396864696af88be1ca12ee0f1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240503 02:15:07.438709  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling UndoDeltaBlockGCOp(c23b06764ce54acb8388de94aea4cd4b): 4581 bytes on disk
I20240503 02:15:07.439316  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: UndoDeltaBlockGCOp(c23b06764ce54acb8388de94aea4cd4b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20240503 02:15:07.469987  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling FlushMRSOp(ecb5c78a22a74728ab40f1da8f90388c): perf score=0.033402
I20240503 02:15:07.505295  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: FlushMRSOp(ecb5c78a22a74728ab40f1da8f90388c) complete. Timing: real 0.035s	user 0.020s	sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":457,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":1669,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":670,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":624,"threads_started":2,"wal-append.queue_time_us":401}
I20240503 02:15:07.506330  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling UndoDeltaBlockGCOp(ecb5c78a22a74728ab40f1da8f90388c): 4923 bytes on disk
I20240503 02:15:07.506930  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: UndoDeltaBlockGCOp(ecb5c78a22a74728ab40f1da8f90388c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240503 02:15:07.572770  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:07.839897  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:07.854287  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:07.854280 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:15:07.982569 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:08.096097  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:08.124043  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 02:15:08.260649  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling FlushMRSOp(4af5884849ec4ac3891ebf0f08ee2cff): perf score=0.033336
I20240503 02:15:08.313248  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: FlushMRSOp(4af5884849ec4ac3891ebf0f08ee2cff) complete. Timing: real 0.052s	user 0.040s	sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4789,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1601,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":702,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":10186,"threads_started":2,"wal-append.queue_time_us":5622}
I20240503 02:15:08.314512  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling UndoDeltaBlockGCOp(4af5884849ec4ac3891ebf0f08ee2cff): 4923 bytes on disk
I20240503 02:15:08.317010  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: UndoDeltaBlockGCOp(4af5884849ec4ac3891ebf0f08ee2cff) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240503 02:15:08.345463  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(72e2bb75da654e8ab45bba8ac077f28c): perf score=0.033362
W20240503 02:15:08.348919  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:08.380182  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(72e2bb75da654e8ab45bba8ac077f28c) complete. Timing: real 0.034s	user 0.028s	sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":329,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1511,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":691,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":472,"threads_started":2,"wal-append.queue_time_us":350}
I20240503 02:15:08.381611  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(4af5884849ec4ac3891ebf0f08ee2cff): perf score=0.033367
I20240503 02:15:08.445117  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling FlushMRSOp(834aaba965c8443c862e98a070e36cac): perf score=0.033390
I20240503 02:15:08.449666  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(4af5884849ec4ac3891ebf0f08ee2cff) complete. Timing: real 0.068s	user 0.044s	sys 0.020s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":93,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1568,"drs_written":1,"lbm_read_time_us":152,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1124,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":8448,"thread_start_us":742,"threads_started":1,"wal-append.queue_time_us":854}
I20240503 02:15:08.451663  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(4af5884849ec4ac3891ebf0f08ee2cff): 4923 bytes on disk
I20240503 02:15:08.452390  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(4af5884849ec4ac3891ebf0f08ee2cff) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240503 02:15:08.454315  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(72e2bb75da654e8ab45bba8ac077f28c): 4583 bytes on disk
I20240503 02:15:08.455217  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(72e2bb75da654e8ab45bba8ac077f28c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20240503 02:15:08.485175  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: FlushMRSOp(834aaba965c8443c862e98a070e36cac) complete. Timing: real 0.040s	user 0.035s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":290,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1446,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":707,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":452,"threads_started":2,"wal-append.queue_time_us":365}
I20240503 02:15:08.497488  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling UndoDeltaBlockGCOp(834aaba965c8443c862e98a070e36cac): 4836 bytes on disk
I20240503 02:15:08.498131  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: UndoDeltaBlockGCOp(834aaba965c8443c862e98a070e36cac) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
W20240503 02:15:08.514223  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:08.735764  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:08.829032  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:09.037796  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:09.065829 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:09.220561 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:09.243714  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:09.460384  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(aeaee03abc644807b8278cfb78e4f02b): perf score=0.033336
I20240503 02:15:09.490514  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(aeaee03abc644807b8278cfb78e4f02b) complete. Timing: real 0.030s	user 0.019s	sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":386,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1509,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":689,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":486,"threads_started":2,"wal-append.queue_time_us":314}
I20240503 02:15:09.491667  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(b64f66ace86e42f3ac213ba85bcab584): perf score=0.033340
I20240503 02:15:09.502789  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling FlushMRSOp(b64f66ace86e42f3ac213ba85bcab584): perf score=0.033345
I20240503 02:15:09.525928  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(b64f66ace86e42f3ac213ba85bcab584) complete. Timing: real 0.034s	user 0.026s	sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":68,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":1722,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":563,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":306,"threads_started":1,"wal-append.queue_time_us":411}
I20240503 02:15:09.527833  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(b64f66ace86e42f3ac213ba85bcab584): 4923 bytes on disk
I20240503 02:15:09.528548  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(b64f66ace86e42f3ac213ba85bcab584) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240503 02:15:09.529963  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(aeaee03abc644807b8278cfb78e4f02b): 4583 bytes on disk
I20240503 02:15:09.530548  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(aeaee03abc644807b8278cfb78e4f02b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240503 02:15:09.538265  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: FlushMRSOp(b64f66ace86e42f3ac213ba85bcab584) complete. Timing: real 0.035s	user 0.024s	sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":340,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":6568,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":722,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":527,"threads_started":2,"wal-append.queue_time_us":385}
I20240503 02:15:09.539696  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling UndoDeltaBlockGCOp(b64f66ace86e42f3ac213ba85bcab584): 4923 bytes on disk
I20240503 02:15:09.540303  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: UndoDeltaBlockGCOp(b64f66ace86e42f3ac213ba85bcab584) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240503 02:15:09.541422  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling FlushMRSOp(aeaee03abc644807b8278cfb78e4f02b): perf score=0.033353
W20240503 02:15:09.542328  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:15:09.580821  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: FlushMRSOp(aeaee03abc644807b8278cfb78e4f02b) complete. Timing: real 0.039s	user 0.027s	sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":81,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":1699,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":672,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":234,"threads_started":1,"wal-append.queue_time_us":332}
I20240503 02:15:09.581894  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling UndoDeltaBlockGCOp(aeaee03abc644807b8278cfb78e4f02b): 4583 bytes on disk
I20240503 02:15:09.582381  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: UndoDeltaBlockGCOp(aeaee03abc644807b8278cfb78e4f02b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240503 02:15:09.593221  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling FlushMRSOp(45fe6ec58eed40f1898d4a2f5727cfa1): perf score=0.033374
I20240503 02:15:09.634596  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: FlushMRSOp(45fe6ec58eed40f1898d4a2f5727cfa1) complete. Timing: real 0.041s	user 0.030s	sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":603,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1637,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":885,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":680,"threads_started":2,"wal-append.queue_time_us":332}
I20240503 02:15:09.636099  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(45fe6ec58eed40f1898d4a2f5727cfa1): 4838 bytes on disk
I20240503 02:15:09.636798  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(45fe6ec58eed40f1898d4a2f5727cfa1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240503 02:15:09.656620  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:09.924196 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:10.117434 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:10.125958  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:10.191943  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:10.367441  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:10.437332 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:10.457844  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:10.798224  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:11.058152  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:11.275218  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:11.490192 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:11.508999  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:11.517053 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:11.736480  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:11.860956  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:11.995862  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:15:12.429281  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:12.448678 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 02:15:12.449915 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:12.468991 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:12.649251  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:12.752375  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:12.904395  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:12.941973 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:15:13.074198  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:13.124272 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:13.144039  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:15:13.383363  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:13.408664  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:13.883615  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:14.061305 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:14.062670  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:14.073179 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:14.141866  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:14.380722  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:14.810444  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:14.871152  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:14.925012 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:14.982249  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:15:14.983381  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:15.029592 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:15.241686  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:15.590288 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:15.628509  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:15.787853  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:16.055595  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:16.097059  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:16.443637  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:16.490270  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:16.598309 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:16.679311  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:16.888365 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:16.909898  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:17.110502  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:15:17.334117  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:17.480077  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:17.493551 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:15:17.512133  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:17.549311 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:17.823789 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:17.919723  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:17.927650 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:18.017473  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:18.223523  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:15:18.241456 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:15:18.397338  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:18.550809  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:18.815584 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:18.948594  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:18.963268  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:19.024463  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:19.095981 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:19.485392  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:19.587827  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:19.953039  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:19.972167 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling FlushMRSOp(5ea56405515a40f785726bd58ec3d26a): perf score=0.033350
I20240503 02:15:19.973085 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling FlushMRSOp(b64f66ace86e42f3ac213ba85bcab584): perf score=0.033357
W20240503 02:15:20.048588 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:15:20.083817 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: FlushMRSOp(b64f66ace86e42f3ac213ba85bcab584) complete. Timing: real 0.110s	user 0.062s	sys 0.043s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":19987,"dirs.queue_time_us":16209,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":1000,"drs_written":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4,"lbm_write_time_us":979,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":34508,"threads_started":3,"wal-append.queue_time_us":8416}
I20240503 02:15:20.086838 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling FlushMRSOp(0f44b2b53de84a87a8757c0d00bfed37): perf score=0.033387
W20240503 02:15:20.146646  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:15:20.147706  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:15:20.174860  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: FlushMRSOp(5ea56405515a40f785726bd58ec3d26a) complete. Timing: real 0.202s	user 0.083s	sys 0.022s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":386,"dirs.queue_time_us":8345,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":4288,"drs_written":1,"lbm_read_time_us":156,"lbm_reads_lt_1ms":4,"lbm_write_time_us":921,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11979,"threads_started":3,"wal-append.queue_time_us":3413}
I20240503 02:15:20.176928 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling UndoDeltaBlockGCOp(5ea56405515a40f785726bd58ec3d26a): 4583 bytes on disk
I20240503 02:15:20.182296  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: UndoDeltaBlockGCOp(5ea56405515a40f785726bd58ec3d26a) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4}
I20240503 02:15:20.207572 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling FlushMRSOp(f14607425da84bb4b162a5789daa844c): perf score=0.033379
I20240503 02:15:20.230507 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: FlushMRSOp(0f44b2b53de84a87a8757c0d00bfed37) complete. Timing: real 0.135s	user 0.047s	sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1601,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1028,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13956,"threads_started":1,"wal-append.queue_time_us":10609}
I20240503 02:15:20.232199 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling UndoDeltaBlockGCOp(b64f66ace86e42f3ac213ba85bcab584): 4923 bytes on disk
I20240503 02:15:20.233037 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: UndoDeltaBlockGCOp(b64f66ace86e42f3ac213ba85bcab584) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 02:15:20.234315 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling UndoDeltaBlockGCOp(0f44b2b53de84a87a8757c0d00bfed37): 4836 bytes on disk
I20240503 02:15:20.234884 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: UndoDeltaBlockGCOp(0f44b2b53de84a87a8757c0d00bfed37) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240503 02:15:20.332109 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:20.343298 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:15:20.372464 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: FlushMRSOp(f14607425da84bb4b162a5789daa844c) complete. Timing: real 0.164s	user 0.080s	sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":13596,"dirs.queue_time_us":19337,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":1123,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1521,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":46267,"threads_started":3,"wal-append.queue_time_us":10086}
I20240503 02:15:20.378757 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling FlushMRSOp(4f2cedba9f9a4e709623fceeffcfd095): perf score=0.033408
W20240503 02:15:20.540128  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:20.550374 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: FlushMRSOp(4f2cedba9f9a4e709623fceeffcfd095) complete. Timing: real 0.169s	user 0.047s	sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":1583,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1004,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2216,"threads_started":1,"wal-append.queue_time_us":2078}
I20240503 02:15:20.566253 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling UndoDeltaBlockGCOp(f14607425da84bb4b162a5789daa844c): 4838 bytes on disk
I20240503 02:15:20.569737 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: UndoDeltaBlockGCOp(f14607425da84bb4b162a5789daa844c) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240503 02:15:20.571245 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling UndoDeltaBlockGCOp(4f2cedba9f9a4e709623fceeffcfd095): 4836 bytes on disk
I20240503 02:15:20.572474 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: UndoDeltaBlockGCOp(4f2cedba9f9a4e709623fceeffcfd095) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
W20240503 02:15:20.640955  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:15:20.738034 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling FlushMRSOp(de9c2a3396864696af88be1ca12ee0f1): perf score=0.033395
I20240503 02:15:20.870448 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: FlushMRSOp(de9c2a3396864696af88be1ca12ee0f1) complete. Timing: real 0.132s	user 0.059s	sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6529,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":6404,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":831,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":10890,"threads_started":2,"wal-append.queue_time_us":1125}
I20240503 02:15:20.879966 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling UndoDeltaBlockGCOp(de9c2a3396864696af88be1ca12ee0f1): 4838 bytes on disk
I20240503 02:15:20.880607 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: UndoDeltaBlockGCOp(de9c2a3396864696af88be1ca12ee0f1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
W20240503 02:15:20.880980  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:15:20.938766 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling FlushMRSOp(80b4e6aa20f64473be8bcbf47baf66e4): perf score=0.033383
I20240503 02:15:21.031029  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: FlushMRSOp(80b4e6aa20f64473be8bcbf47baf66e4) complete. Timing: real 0.092s	user 0.059s	sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8667,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":16625,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":901,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":18080,"threads_started":2,"wal-append.queue_time_us":9347}
I20240503 02:15:21.032542 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling UndoDeltaBlockGCOp(80b4e6aa20f64473be8bcbf47baf66e4): 4923 bytes on disk
I20240503 02:15:21.033210  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: UndoDeltaBlockGCOp(80b4e6aa20f64473be8bcbf47baf66e4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240503 02:15:21.052834  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:21.264549 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:21.309365  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:15:21.327819 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling FlushMRSOp(d3d4b912095244de816d3089a31b8649): perf score=0.033373
I20240503 02:15:21.366676 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: FlushMRSOp(d3d4b912095244de816d3089a31b8649) complete. Timing: real 0.039s	user 0.038s	sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":420,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1521,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":558,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":501,"threads_started":2,"wal-append.queue_time_us":277}
I20240503 02:15:21.367820 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling FlushMRSOp(7e4acc8b5ceb4cd5b0f3708ff4f19298): perf score=0.033357
I20240503 02:15:21.404286 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: FlushMRSOp(7e4acc8b5ceb4cd5b0f3708ff4f19298) complete. Timing: real 0.036s	user 0.022s	sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1952,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":699,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":346,"threads_started":1,"wal-append.queue_time_us":456}
I20240503 02:15:21.405551 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling FlushMRSOp(aeaee03abc644807b8278cfb78e4f02b): perf score=0.033357
I20240503 02:15:21.439647 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: FlushMRSOp(aeaee03abc644807b8278cfb78e4f02b) complete. Timing: real 0.033s	user 0.029s	sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":77,"dirs.run_cpu_time_us":261,"dirs.run_wall_time_us":1776,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":682,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":214,"threads_started":1,"wal-append.queue_time_us":371}
I20240503 02:15:21.440656 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling UndoDeltaBlockGCOp(7e4acc8b5ceb4cd5b0f3708ff4f19298): 4838 bytes on disk
I20240503 02:15:21.441206 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: UndoDeltaBlockGCOp(7e4acc8b5ceb4cd5b0f3708ff4f19298) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 02:15:21.441973 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling UndoDeltaBlockGCOp(d3d4b912095244de816d3089a31b8649): 4838 bytes on disk
I20240503 02:15:21.442394 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: UndoDeltaBlockGCOp(d3d4b912095244de816d3089a31b8649) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 02:15:21.443127 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling UndoDeltaBlockGCOp(aeaee03abc644807b8278cfb78e4f02b): 4583 bytes on disk
I20240503 02:15:21.443524 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: UndoDeltaBlockGCOp(aeaee03abc644807b8278cfb78e4f02b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240503 02:15:21.452788  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:21.506630  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:21.630717 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:15:21.634996 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling FlushMRSOp(3dbfc7f12fc142f98b1d1c026fd49f5a): perf score=0.033378
I20240503 02:15:21.695770 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: FlushMRSOp(3dbfc7f12fc142f98b1d1c026fd49f5a) complete. Timing: real 0.061s	user 0.045s	sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2697,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1561,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2120,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3062,"threads_started":2,"wal-append.queue_time_us":294}
I20240503 02:15:21.696736 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling UndoDeltaBlockGCOp(3dbfc7f12fc142f98b1d1c026fd49f5a): 4583 bytes on disk
I20240503 02:15:21.697223 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: UndoDeltaBlockGCOp(3dbfc7f12fc142f98b1d1c026fd49f5a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 02:15:21.788832 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling FlushMRSOp(32e527fa9b2f441cb5668435681b4ff6): perf score=0.033390
I20240503 02:15:21.828972  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: FlushMRSOp(32e527fa9b2f441cb5668435681b4ff6) complete. Timing: real 0.040s	user 0.037s	sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":374,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":1768,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":597,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":466,"threads_started":2,"wal-append.queue_time_us":346}
I20240503 02:15:21.830109 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling UndoDeltaBlockGCOp(32e527fa9b2f441cb5668435681b4ff6): 4923 bytes on disk
I20240503 02:15:21.830734  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: UndoDeltaBlockGCOp(32e527fa9b2f441cb5668435681b4ff6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240503 02:15:21.852332  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:22.082558 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling FlushMRSOp(cab70b62bdc747a9b653bffde680f37e): perf score=0.033396
I20240503 02:15:22.116245  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: FlushMRSOp(cab70b62bdc747a9b653bffde680f37e) complete. Timing: real 0.033s	user 0.019s	sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1485,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":577,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":238,"threads_started":1,"wal-append.queue_time_us":366}
I20240503 02:15:22.117360 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling FlushMRSOp(4af5884849ec4ac3891ebf0f08ee2cff): perf score=0.033384
I20240503 02:15:22.183439  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: FlushMRSOp(4af5884849ec4ac3891ebf0f08ee2cff) complete. Timing: real 0.066s	user 0.038s	sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":74,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1620,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":731,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":21455,"threads_started":1,"wal-append.queue_time_us":21556}
I20240503 02:15:22.184691 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling UndoDeltaBlockGCOp(4af5884849ec4ac3891ebf0f08ee2cff): 4923 bytes on disk
I20240503 02:15:22.185199  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: UndoDeltaBlockGCOp(4af5884849ec4ac3891ebf0f08ee2cff) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 02:15:22.186514 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling UndoDeltaBlockGCOp(cab70b62bdc747a9b653bffde680f37e): 4923 bytes on disk
I20240503 02:15:22.189858  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: UndoDeltaBlockGCOp(cab70b62bdc747a9b653bffde680f37e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
W20240503 02:15:22.263948  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:22.309962  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:15:22.310854  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:22.443184 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling FlushMRSOp(f231fe5dfd9e4c87b90bad75818a7060): perf score=0.033371
I20240503 02:15:22.451496 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling FlushMRSOp(7e966caa651e4e42890be5d79f851a1f): perf score=0.033339
W20240503 02:15:22.498643 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:15:22.522753  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: FlushMRSOp(f231fe5dfd9e4c87b90bad75818a7060) complete. Timing: real 0.079s	user 0.053s	sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1918,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1147,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":336,"threads_started":1,"wal-append.queue_time_us":483}
I20240503 02:15:22.524216 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling UndoDeltaBlockGCOp(f231fe5dfd9e4c87b90bad75818a7060): 4583 bytes on disk
I20240503 02:15:22.524760  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: UndoDeltaBlockGCOp(f231fe5dfd9e4c87b90bad75818a7060) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 02:15:22.526335 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: FlushMRSOp(7e966caa651e4e42890be5d79f851a1f) complete. Timing: real 0.074s	user 0.044s	sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":7210,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":8052,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1066,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7907,"threads_started":2,"wal-append.queue_time_us":715}
I20240503 02:15:22.527519 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling UndoDeltaBlockGCOp(7e966caa651e4e42890be5d79f851a1f): 4919 bytes on disk
I20240503 02:15:22.528033 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: UndoDeltaBlockGCOp(7e966caa651e4e42890be5d79f851a1f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240503 02:15:22.674862 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:22.688205  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:15:22.699304 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling FlushMRSOp(72e2bb75da654e8ab45bba8ac077f28c): perf score=0.033360
I20240503 02:15:22.763995 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: FlushMRSOp(72e2bb75da654e8ab45bba8ac077f28c) complete. Timing: real 0.064s	user 0.041s	sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":307,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":7021,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":775,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":15119,"threads_started":2,"wal-append.queue_time_us":14774}
W20240503 02:15:22.765416 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:15:22.766187 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling UndoDeltaBlockGCOp(72e2bb75da654e8ab45bba8ac077f28c): 4583 bytes on disk
I20240503 02:15:22.768774 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: UndoDeltaBlockGCOp(72e2bb75da654e8ab45bba8ac077f28c) complete. Timing: real 0.002s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1533,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240503 02:15:22.771057 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling FlushMRSOp(5ba1f505010b403c91d3870cee19952a): perf score=0.033354
I20240503 02:15:22.776983 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling FlushMRSOp(c23b06764ce54acb8388de94aea4cd4b): perf score=0.033347
I20240503 02:15:22.783850 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling FlushMRSOp(c45c5a14b85447c78fe2a27be4392c88): perf score=0.033348
W20240503 02:15:22.786896 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:15:22.851976 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: FlushMRSOp(5ba1f505010b403c91d3870cee19952a) complete. Timing: real 0.081s	user 0.028s	sys 0.026s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4111,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":1914,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":803,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":8584,"threads_started":1,"wal-append.queue_time_us":8702}
I20240503 02:15:22.853889 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling UndoDeltaBlockGCOp(5ba1f505010b403c91d3870cee19952a): 4583 bytes on disk
I20240503 02:15:22.854655 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: UndoDeltaBlockGCOp(5ba1f505010b403c91d3870cee19952a) complete. Timing: real 0.001s	user 0.000s	sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":134,"lbm_reads_lt_1ms":4}
I20240503 02:15:22.860397 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: FlushMRSOp(c45c5a14b85447c78fe2a27be4392c88) complete. Timing: real 0.076s	user 0.045s	sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":2929,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":848,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":714,"threads_started":1,"wal-append.queue_time_us":861}
I20240503 02:15:22.861723 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling UndoDeltaBlockGCOp(c45c5a14b85447c78fe2a27be4392c88): 4923 bytes on disk
I20240503 02:15:22.863966 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: UndoDeltaBlockGCOp(c45c5a14b85447c78fe2a27be4392c88) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 02:15:22.864784  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: FlushMRSOp(c23b06764ce54acb8388de94aea4cd4b) complete. Timing: real 0.086s	user 0.028s	sys 0.028s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1374,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":952,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":4820,"threads_started":1,"wal-append.queue_time_us":9101}
I20240503 02:15:22.865890 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling UndoDeltaBlockGCOp(c23b06764ce54acb8388de94aea4cd4b): 4581 bytes on disk
I20240503 02:15:22.866348  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: UndoDeltaBlockGCOp(c23b06764ce54acb8388de94aea4cd4b) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240503 02:15:22.890291  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:23.110713  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:23.316388 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:15:23.321731  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:23.328198  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 02:15:23.368783 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling FlushMRSOp(45fe6ec58eed40f1898d4a2f5727cfa1): perf score=0.033377
I20240503 02:15:23.402863  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: FlushMRSOp(45fe6ec58eed40f1898d4a2f5727cfa1) complete. Timing: real 0.034s	user 0.022s	sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":328,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":1655,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":644,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":426,"threads_started":2,"wal-append.queue_time_us":317}
W20240503 02:15:23.403875  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:15:23.404251 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling UndoDeltaBlockGCOp(45fe6ec58eed40f1898d4a2f5727cfa1): 4838 bytes on disk
I20240503 02:15:23.404801  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: UndoDeltaBlockGCOp(45fe6ec58eed40f1898d4a2f5727cfa1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240503 02:15:23.781739  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:23.787792 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:23.828099  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:23.912528 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:23.938480  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:24.422336  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:24.797960  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:24.968497 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:25.086671  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:25.187189  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:15:25.188026  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:25.196851 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:25.294262 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:25.304006  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:25.548338  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:25.751407  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:26.116854 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling FlushMRSOp(de9c2a3396864696af88be1ca12ee0f1): perf score=0.033376
W20240503 02:15:26.127485  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:15:26.183028 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: FlushMRSOp(de9c2a3396864696af88be1ca12ee0f1) complete. Timing: real 0.066s	user 0.047s	sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":314,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1637,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":785,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":391,"threads_started":2,"wal-append.queue_time_us":294}
I20240503 02:15:26.184105 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling UndoDeltaBlockGCOp(de9c2a3396864696af88be1ca12ee0f1): 4838 bytes on disk
I20240503 02:15:26.184583 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: UndoDeltaBlockGCOp(de9c2a3396864696af88be1ca12ee0f1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240503 02:15:26.223276  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:26.350224  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:26.470994 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:26.483180  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:26.544298 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:15:26.686975 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling FlushMRSOp(ecb5c78a22a74728ab40f1da8f90388c): perf score=0.033396
I20240503 02:15:26.733940 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: FlushMRSOp(ecb5c78a22a74728ab40f1da8f90388c) complete. Timing: real 0.047s	user 0.031s	sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":417,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1544,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2449,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":713,"threads_started":2,"wal-append.queue_time_us":551}
I20240503 02:15:26.735486 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling UndoDeltaBlockGCOp(ecb5c78a22a74728ab40f1da8f90388c): 4923 bytes on disk
I20240503 02:15:26.736213 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: UndoDeltaBlockGCOp(ecb5c78a22a74728ab40f1da8f90388c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
W20240503 02:15:26.845911  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:26.878274 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling FlushMRSOp(834aaba965c8443c862e98a070e36cac): perf score=0.033354
I20240503 02:15:26.908565 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: FlushMRSOp(834aaba965c8443c862e98a070e36cac) complete. Timing: real 0.030s	user 0.020s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":353,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":1763,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":624,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":421,"threads_started":2,"wal-append.queue_time_us":300}
I20240503 02:15:26.909664 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling UndoDeltaBlockGCOp(834aaba965c8443c862e98a070e36cac): 4836 bytes on disk
I20240503 02:15:26.910219 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: UndoDeltaBlockGCOp(834aaba965c8443c862e98a070e36cac) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 02:15:26.911027 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling FlushMRSOp(f14607425da84bb4b162a5789daa844c): perf score=0.033363
I20240503 02:15:26.946630 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: FlushMRSOp(f14607425da84bb4b162a5789daa844c) complete. Timing: real 0.035s	user 0.027s	sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":296,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":3414,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":598,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":226,"threads_started":1,"wal-append.queue_time_us":337}
I20240503 02:15:26.948158 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling UndoDeltaBlockGCOp(f14607425da84bb4b162a5789daa844c): 4838 bytes on disk
I20240503 02:15:26.948777 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: UndoDeltaBlockGCOp(f14607425da84bb4b162a5789daa844c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240503 02:15:27.371596  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:15:27.384521  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:27.490234 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling FlushMRSOp(402de43677c44deab8793da1895d4fb2): perf score=0.033341
W20240503 02:15:27.504129  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:27.522008  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:15:27.526021 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: FlushMRSOp(402de43677c44deab8793da1895d4fb2) complete. Timing: real 0.036s	user 0.029s	sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":296,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1709,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":696,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":416,"threads_started":2,"wal-append.queue_time_us":318}
I20240503 02:15:27.527099 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling UndoDeltaBlockGCOp(402de43677c44deab8793da1895d4fb2): 4838 bytes on disk
I20240503 02:15:27.527611 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: UndoDeltaBlockGCOp(402de43677c44deab8793da1895d4fb2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240503 02:15:27.683817 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:15:27.703104 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:27.778184  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:27.839715 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:15:27.921327 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling FlushMRSOp(0f44b2b53de84a87a8757c0d00bfed37): perf score=0.033387
I20240503 02:15:27.976338  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: FlushMRSOp(0f44b2b53de84a87a8757c0d00bfed37) complete. Timing: real 0.055s	user 0.038s	sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":347,"dirs.run_cpu_time_us":306,"dirs.run_wall_time_us":3042,"drs_written":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4,"lbm_write_time_us":804,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6145,"threads_started":2,"wal-append.queue_time_us":431}
I20240503 02:15:27.977957 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling UndoDeltaBlockGCOp(0f44b2b53de84a87a8757c0d00bfed37): 4836 bytes on disk
I20240503 02:15:27.978700  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: UndoDeltaBlockGCOp(0f44b2b53de84a87a8757c0d00bfed37) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":152,"lbm_reads_lt_1ms":4}
W20240503 02:15:27.998085  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:28.011121 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:15:28.205304 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling FlushMRSOp(007b59de142c4b95966494ba097b1a09): perf score=0.033387
I20240503 02:15:28.231104 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling FlushMRSOp(4f2cedba9f9a4e709623fceeffcfd095): perf score=0.033370
I20240503 02:15:28.262320 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: FlushMRSOp(007b59de142c4b95966494ba097b1a09) complete. Timing: real 0.057s	user 0.035s	sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1044,"dirs.run_cpu_time_us":258,"dirs.run_wall_time_us":1623,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":860,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1985,"threads_started":2,"wal-append.queue_time_us":1188}
I20240503 02:15:28.263806 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling UndoDeltaBlockGCOp(007b59de142c4b95966494ba097b1a09): 4921 bytes on disk
I20240503 02:15:28.264413 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: UndoDeltaBlockGCOp(007b59de142c4b95966494ba097b1a09) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 02:15:28.299739  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: FlushMRSOp(4f2cedba9f9a4e709623fceeffcfd095) complete. Timing: real 0.068s	user 0.048s	sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":65,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1572,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":800,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":226,"threads_started":1,"wal-append.queue_time_us":343}
I20240503 02:15:28.301267 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling UndoDeltaBlockGCOp(4f2cedba9f9a4e709623fceeffcfd095): 4836 bytes on disk
I20240503 02:15:28.301942  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: UndoDeltaBlockGCOp(4f2cedba9f9a4e709623fceeffcfd095) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240503 02:15:28.310523  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:28.392923 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:15:28.434813  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:15:28.727766  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:28.767465  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:28.921281  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:29.020849  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:15:29.055421 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling FlushMRSOp(72e2bb75da654e8ab45bba8ac077f28c): perf score=0.033368
I20240503 02:15:29.098253  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: FlushMRSOp(72e2bb75da654e8ab45bba8ac077f28c) complete. Timing: real 0.043s	user 0.035s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":392,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":1674,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":677,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":2977,"threads_started":2,"wal-append.queue_time_us":2833}
I20240503 02:15:29.099900 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling UndoDeltaBlockGCOp(72e2bb75da654e8ab45bba8ac077f28c): 4583 bytes on disk
I20240503 02:15:29.100575  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: UndoDeltaBlockGCOp(72e2bb75da654e8ab45bba8ac077f28c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240503 02:15:29.105698 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:29.154542  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:29.268425 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling FlushMRSOp(2d69c71d9dfd47a1a2429e7caf5e33ac): perf score=0.033335
W20240503 02:15:29.283269 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:15:29.298550 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: FlushMRSOp(2d69c71d9dfd47a1a2429e7caf5e33ac) complete. Timing: real 0.030s	user 0.019s	sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":355,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1666,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":642,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":415,"threads_started":2,"wal-append.queue_time_us":326}
I20240503 02:15:29.299968 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling UndoDeltaBlockGCOp(2d69c71d9dfd47a1a2429e7caf5e33ac): 4583 bytes on disk
I20240503 02:15:29.300565 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: UndoDeltaBlockGCOp(2d69c71d9dfd47a1a2429e7caf5e33ac) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240503 02:15:29.535529 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling FlushMRSOp(6043da7d2e584af885ff6a5bf8f06912): perf score=0.033400
I20240503 02:15:29.571463 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: FlushMRSOp(6043da7d2e584af885ff6a5bf8f06912) complete. Timing: real 0.036s	user 0.024s	sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":440,"dirs.run_cpu_time_us":302,"dirs.run_wall_time_us":1719,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":751,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":549,"threads_started":2,"wal-append.queue_time_us":382}
I20240503 02:15:29.572690 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling UndoDeltaBlockGCOp(6043da7d2e584af885ff6a5bf8f06912): 4583 bytes on disk
I20240503 02:15:29.573238 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: UndoDeltaBlockGCOp(6043da7d2e584af885ff6a5bf8f06912) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
W20240503 02:15:29.589988  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:30.039350  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:15:30.076069 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling FlushMRSOp(8a8affdde4d8417f95f0585dd01e7d59): perf score=0.033390
I20240503 02:15:30.108304 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: FlushMRSOp(8a8affdde4d8417f95f0585dd01e7d59) complete. Timing: real 0.032s	user 0.018s	sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":276,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1711,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":603,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":396,"threads_started":2,"wal-append.queue_time_us":313}
I20240503 02:15:30.109326 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling UndoDeltaBlockGCOp(8a8affdde4d8417f95f0585dd01e7d59): 4838 bytes on disk
I20240503 02:15:30.109841 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: UndoDeltaBlockGCOp(8a8affdde4d8417f95f0585dd01e7d59) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240503 02:15:30.207969  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:30.208420  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:30.305352 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling FlushMRSOp(5ea56405515a40f785726bd58ec3d26a): perf score=0.033361
W20240503 02:15:30.333551 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:15:30.337612 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: FlushMRSOp(5ea56405515a40f785726bd58ec3d26a) complete. Timing: real 0.032s	user 0.026s	sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":313,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":1676,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":606,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":404,"threads_started":2,"wal-append.queue_time_us":278}
I20240503 02:15:30.338749 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling UndoDeltaBlockGCOp(5ea56405515a40f785726bd58ec3d26a): 4583 bytes on disk
I20240503 02:15:30.339274 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: UndoDeltaBlockGCOp(5ea56405515a40f785726bd58ec3d26a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240503 02:15:30.434698  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:15:30.503238  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:30.550695 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:30.595016 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:15:30.607548 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling FlushMRSOp(d3d4b912095244de816d3089a31b8649): perf score=0.033379
I20240503 02:15:30.650081  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: FlushMRSOp(d3d4b912095244de816d3089a31b8649) complete. Timing: real 0.042s	user 0.026s	sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":312,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1674,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":806,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":465,"threads_started":2,"wal-append.queue_time_us":348}
I20240503 02:15:30.652663 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling UndoDeltaBlockGCOp(d3d4b912095244de816d3089a31b8649): 4838 bytes on disk
W20240503 02:15:30.653681  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:30.654470  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: UndoDeltaBlockGCOp(d3d4b912095244de816d3089a31b8649) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4}
W20240503 02:15:31.224913  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:31.410048 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling FlushMRSOp(6043da7d2e584af885ff6a5bf8f06912): perf score=0.033339
W20240503 02:15:31.418260  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:15:31.451576  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: FlushMRSOp(6043da7d2e584af885ff6a5bf8f06912) complete. Timing: real 0.041s	user 0.028s	sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6444,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1923,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":944,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6804,"threads_started":2,"wal-append.queue_time_us":321}
I20240503 02:15:31.452678 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling UndoDeltaBlockGCOp(6043da7d2e584af885ff6a5bf8f06912): 4583 bytes on disk
I20240503 02:15:31.453171  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: UndoDeltaBlockGCOp(6043da7d2e584af885ff6a5bf8f06912) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240503 02:15:31.609966  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:31.663740 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:31.706631  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:31.849807  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:31.858673 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:15:31.922024  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling FlushMRSOp(c45c5a14b85447c78fe2a27be4392c88): perf score=0.033378
I20240503 02:15:31.957260  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: FlushMRSOp(c45c5a14b85447c78fe2a27be4392c88) complete. Timing: real 0.035s	user 0.027s	sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":334,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1583,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2830,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":445,"threads_started":2,"wal-append.queue_time_us":342}
I20240503 02:15:31.958317  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling UndoDeltaBlockGCOp(c45c5a14b85447c78fe2a27be4392c88): 4923 bytes on disk
I20240503 02:15:31.958964  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: UndoDeltaBlockGCOp(c45c5a14b85447c78fe2a27be4392c88) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
W20240503 02:15:32.251876  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:32.389997  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:15:32.440939  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:32.635033  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:32.721635  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:32.753867 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:15:32.999125  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:33.124543 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:33.132704 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:33.140472  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:33.265872 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:15:33.349925 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling FlushMRSOp(cab70b62bdc747a9b653bffde680f37e): perf score=0.033389
I20240503 02:15:33.405383 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: FlushMRSOp(cab70b62bdc747a9b653bffde680f37e) complete. Timing: real 0.055s	user 0.044s	sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":352,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1726,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":625,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":449,"threads_started":2,"wal-append.queue_time_us":324}
I20240503 02:15:33.407078 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling UndoDeltaBlockGCOp(cab70b62bdc747a9b653bffde680f37e): 4923 bytes on disk
I20240503 02:15:33.408198 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: UndoDeltaBlockGCOp(cab70b62bdc747a9b653bffde680f37e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240503 02:15:33.624298  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(2d69c71d9dfd47a1a2429e7caf5e33ac): perf score=0.033394
I20240503 02:15:33.677732  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(2d69c71d9dfd47a1a2429e7caf5e33ac) complete. Timing: real 0.053s	user 0.044s	sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":697,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1504,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":890,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6400,"threads_started":2,"wal-append.queue_time_us":6255}
I20240503 02:15:33.680022  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(2d69c71d9dfd47a1a2429e7caf5e33ac): 4583 bytes on disk
I20240503 02:15:33.680740  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(2d69c71d9dfd47a1a2429e7caf5e33ac) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
W20240503 02:15:33.683938 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:15:33.693632  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:33.802323  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 02:15:33.985124  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling FlushMRSOp(834aaba965c8443c862e98a070e36cac): perf score=0.033361
W20240503 02:15:33.998483  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:15:34.018129  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: FlushMRSOp(834aaba965c8443c862e98a070e36cac) complete. Timing: real 0.033s	user 0.022s	sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":338,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":1580,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":630,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":464,"threads_started":2,"wal-append.queue_time_us":342}
I20240503 02:15:34.019227  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(834aaba965c8443c862e98a070e36cac): 4836 bytes on disk
I20240503 02:15:34.019745  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(834aaba965c8443c862e98a070e36cac) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240503 02:15:34.082105  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:34.251523  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:34.338074 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:34.352351 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:34.431025  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:34.472160  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:34.977084  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:35.099426  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:35.401957  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:35.467540 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:35.486204  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:15:35.521423  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:15:35.663115  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling FlushMRSOp(45fe6ec58eed40f1898d4a2f5727cfa1): perf score=0.033350
I20240503 02:15:35.690574  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(2448e12efbd44a90b1f8bce9519e73e0): perf score=0.033379
W20240503 02:15:35.724031  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:15:35.726382  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: FlushMRSOp(45fe6ec58eed40f1898d4a2f5727cfa1) complete. Timing: real 0.063s	user 0.052s	sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":440,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1048,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":685,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":582,"threads_started":2,"wal-append.queue_time_us":311}
I20240503 02:15:35.727337  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling UndoDeltaBlockGCOp(45fe6ec58eed40f1898d4a2f5727cfa1): 4838 bytes on disk
I20240503 02:15:35.727979  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: UndoDeltaBlockGCOp(45fe6ec58eed40f1898d4a2f5727cfa1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240503 02:15:35.736402  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(2448e12efbd44a90b1f8bce9519e73e0) complete. Timing: real 0.045s	user 0.032s	sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":439,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":963,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":928,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":547,"threads_started":2,"wal-append.queue_time_us":319}
I20240503 02:15:35.737664  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(2448e12efbd44a90b1f8bce9519e73e0): 4581 bytes on disk
I20240503 02:15:35.738171  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(2448e12efbd44a90b1f8bce9519e73e0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240503 02:15:35.754032 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:35.831720 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:35.973933  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:36.147620  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:15:36.240959  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling FlushMRSOp(80b4e6aa20f64473be8bcbf47baf66e4): perf score=0.033369
I20240503 02:15:36.274050  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: FlushMRSOp(80b4e6aa20f64473be8bcbf47baf66e4) complete. Timing: real 0.033s	user 0.019s	sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":299,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1417,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":643,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":398,"threads_started":2,"wal-append.queue_time_us":302}
I20240503 02:15:36.275249  8148 maintenance_manager.cc:419] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Scheduling UndoDeltaBlockGCOp(80b4e6aa20f64473be8bcbf47baf66e4): 4923 bytes on disk
I20240503 02:15:36.275830  8039 maintenance_manager.cc:643] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: UndoDeltaBlockGCOp(80b4e6aa20f64473be8bcbf47baf66e4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240503 02:15:36.278158  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling FlushMRSOp(4af5884849ec4ac3891ebf0f08ee2cff): perf score=0.033369
I20240503 02:15:36.327658  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: FlushMRSOp(4af5884849ec4ac3891ebf0f08ee2cff) complete. Timing: real 0.049s	user 0.040s	sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1614,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":969,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":827,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5926,"threads_started":2,"wal-append.queue_time_us":5046}
I20240503 02:15:36.329010  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(4af5884849ec4ac3891ebf0f08ee2cff): 4923 bytes on disk
I20240503 02:15:36.329591  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(4af5884849ec4ac3891ebf0f08ee2cff) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
W20240503 02:15:36.495699  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:36.661532 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:36.724432  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:36.898095  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:36.954834 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:37.067570  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:37.420028  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:15:37.442075  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:37.570799  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:37.932503 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:15:37.991721  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:37.994258 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:38.066887  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:38.158140  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:38.158421 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:38.231623  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:38.452666 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:38.517846  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:38.997593  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:15:39.047787 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:15:39.097740  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:39.103766 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:39.218838  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:39.309067  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:39.482300 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:39.530905  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:40.046535  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:40.094884  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:40.524767  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:40.570115  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:40.570251  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:40.619414  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:15:40.710119 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:40.788573  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:40.860345 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:40.875599 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:40.966748  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:41.391151  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:41.508881 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:41.705029  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:41.856694  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:41.922714  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:41.977260 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:42.373092  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:42.527437  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:42.568787  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 02:15:42.842450  7817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:15:42.843813  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 80b4e6aa20f64473be8bcbf47baf66e4: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:42.918999  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:42.942790 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:15:42.960500  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:42.991461 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:43.062179  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:43.130048  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:43.254230 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:43.309459 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:43.357832  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:43.963552  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:44.013578  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:15:44.014430  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:44.022118 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:44.206925 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:44.254395  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:44.318279 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:15:44.404771  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:44.718364  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:44.899600  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:45.165881 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:45.213245  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:45.523679  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:45.573818  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:45.765197 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:45.795053  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:45.816908 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:45.837584  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:15:46.127290  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:46.382232 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:46.527362  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:46.642820  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:46.690508  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:46.872521 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:47.100036  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:47.217301  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:47.543639  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:47.775557  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:15:47.776552  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:47.787588 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:47.932884  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:47.992339 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:15:48.081013 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:48.081182  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:48.465068  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:48.465178  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:48.733234 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:48.857841  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:49.005914 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:49.080753  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:49.146821 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:49.310011  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:15:49.312366  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:49.382182 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:15:49.678799  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:49.799867  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:50.234429  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:50.296984  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:50.410394 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:50.576267  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:50.625144 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:50.644410  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:50.985949  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:51.145236  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:15:51.208947 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:51.266345  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:51.499600 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:51.585805  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:51.633297 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:51.711465  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:51.794497  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:52.111562  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:52.161027  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:52.586890  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:52.779184  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:15:52.780831  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:52.789233 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:52.907943  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:52.922032 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:53.088904 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:15:53.164670  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:53.486676  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:53.687376 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:53.744129  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:53.923125 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:54.051259  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:54.059441  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:54.084326 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:54.328177  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:54.395134 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:15:54.461194  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:15:54.734874  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:54.894017  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:55.300053  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:55.322271  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:55.427060  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:55.448602 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:55.615656 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:55.745059  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:55.791884  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:56.188279  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:56.278579 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:56.299779  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:15:56.350080  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:56.352353  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:56.573810 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:56.699885 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:56.797288  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:57.194520  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:57.403229  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:57.612635  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:57.879704  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:15:57.899513  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:58.073837 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:58.086490 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:58.158746 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:15:58.172420  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:58.396347  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:58.479959  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:58.631245 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:58.785365  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:58.941275  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:59.207505  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:59.224480 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:59.371311 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:59.431246 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:15:59.496438  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:15:59.522251  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:00.035077  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:16:00.118058  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:16:00.119118  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:00.360034 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:16:00.494544  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:00.669947 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:16:00.942971 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:16:00.957813  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:16:01.056150  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:16:01.066124  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:01.314685  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:16:01.538656  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:16:01.576169  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:01.632220 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:16:01.691946  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:16:01.883854 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:16:02.102173  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:02.292850  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:16:02.567343  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:02.607409  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:02.657744 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:02.957098  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:16:02.957962  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:03.070178 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:03.313436 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:16:03.340332 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:03.412531  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:03.493223  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:03.511474  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:03.860775  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:04.031443  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:04.149504  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:04.342542 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:04.390290 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:04.584110  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:04.627385 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:16:04.778053  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:04.779006  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:04.912153 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:04.993393  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:05.139899  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:05.537264  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:05.853646 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:05.863809 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:06.064493  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:06.154587  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:06.196791  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:06.240000  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:06.515287  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:06.742084  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:06.897816 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:06.946117 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:06.985877  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:16:07.245062  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:07.337145 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:07.495952  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:07.569554  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:07.715308  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:08.171381 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:08.185417  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:08.211505 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:08.235085  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:08.403932 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:16:08.591137  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:08.656808  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:08.716109  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:08.829660  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:08.879503  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:09.261070  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:09.275086 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:09.530107 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:09.782331 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:16:09.794024  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:09.861419  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:16:09.898903  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:10.093490  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:10.099076 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:10.328807  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:10.563282 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:10.786942 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:10.896303  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:10.940544  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:11.199477  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:11.295760  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:11.378484  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:11.426162  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:11.727945 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:11.971781  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:11.980998 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:12.149755  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:16:12.482867  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:12.505108  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:12.618580 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:12.693248  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:13.076710  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:13.099535 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:13.150995 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:13.278339  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:16:13.279234  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:13.446924 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:16:13.515730  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:13.548767  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:13.558732  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:13.946233  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:14.118716 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:14.135963  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:14.315093 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:14.414480  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:14.797266 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:16:14.923089  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:14.923915  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:14.924717  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:15.095809 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:15.263865  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:15.359373  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:15.567909  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:15.604615 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:15.656685 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:15.763294  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:15.836416  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:16.147686  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:16.329409  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:16.588812 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:16.593852  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:16.880772 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:16.890067  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:17.304292  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:16:17.330900  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:17.387423 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:17.524215  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:17.723137  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:17.890096  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:18.098456  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:18.118655  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:18.286247 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:18.359337  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:16:18.360257  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:18.360899 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:18.473199 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:16:18.508780  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:18.660930  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:19.022344 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:19.028327  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:19.148835 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:19.209842  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:19.718899  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:19.925786 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:19.926707 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:20.056038  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:16:20.257227  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:20.301954  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:20.465333  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:20.730315  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:20.746592  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:20.777967 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:20.876816 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:20.899314  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:20.980155  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:21.411546 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:21.427155  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:21.486817  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:21.943403 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:22.079890  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:22.548441  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:16:22.656461  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:22.725843 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:22.814988  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:22.932583  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:23.180776  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:23.196274  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:23.478247 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240503 02:16:23.490868 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:23.502398 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:23.507511  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:23.544250  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:16:23.545368  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:23.697933  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:24.107563  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:24.146215  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:24.146351 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:24.491835 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:24.712819  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:24.933835 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240503 02:16:25.083362  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:25.084219  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:25.131281  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:25.346053 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:25.425662  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:25.536008  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:25.990496  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:26.014791  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:26.050410 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:26.086969  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:26.154233  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:26.162899 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:26.384346 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:26.469624  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:26.678687  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:27.080819  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:27.160815 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:27.233780  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:27.538272  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:27.615665 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:28.065495  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:28.094182  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:16:28.095182  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:28.261289 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:28.442787  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:28.477051  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:28.519392 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:16:28.542330  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:28.592209  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:16:28.592999  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:28.657310 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:28.811555 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:28.812398  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:28.987818  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:29.499939  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:29.548987 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:29.755832  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:29.967065  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:30.151178  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:16:30.223423 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:30.372684 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:30.433403  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:30.822700  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:30.833438  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:30.949724  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:30.980573 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:31.018149  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:31.111111  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:31.136906 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:31.355912  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:31.365707  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:31.562670 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:31.988641  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:32.075774 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:32.455920  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:32.475226  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:32.915061  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:33.113740 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:33.328687  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:33.386528  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:33.528759  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:33.543918  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:16:33.545109  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:33.616335 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240503 02:16:33.618166  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:33.665539 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:33.737931 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:33.858527  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:33.859429  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:33.896471  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:34.072372 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:34.407666  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:34.873255 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:34.916025  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:34.957080  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:35.334619  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:35.341235  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:35.549146 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:35.550143 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:35.928481  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:35.935633  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:35.976682  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:36.020489  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:36.221848 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:36.270385 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:36.277926  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:36.446666  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:36.539443  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:36.552443 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:37.051851 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:37.084177  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:37.218948  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:37.636407  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:38.084463  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:38.279479  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:38.340932  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:38.412588 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:38.490054  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:38.548645  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:16:38.549522  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:38.677482 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:16:38.678644 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:38.689545  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:38.917590 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:39.098268  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:39.099138  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:39.172703 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:39.237495  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:39.318885 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:39.483476  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:39.652886  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:40.050884  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:40.489761  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:40.639662  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:40.676448 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:16:40.713034  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:40.838567 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:40.894472  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:40.906216 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:40.923862  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:41.053740  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:41.182919  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:41.716733 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:41.774549  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:41.805874  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:41.807018 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:41.865123 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:42.177165  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:42.389276  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:42.812070  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:42.862810  7817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:16:42.864203  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 80b4e6aa20f64473be8bcbf47baf66e4: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:43.186717  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:43.205435  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:43.319998 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:43.401109  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:43.432061  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:43.472913  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:43.526410 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:43.621045  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:43.721102 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:16:43.895918  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:16:44.124917 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:44.231122  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:44.232033  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:44.281584 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:44.335183  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:44.426012 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:44.617254  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:44.733160  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:45.031855  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:45.458875  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:45.746235 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:16:45.849710 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:45.863265 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:45.878201  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:45.943526  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:45.954468  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:46.003835  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:46.029891  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:46.057252  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:46.387048 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:46.609880  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:46.644539 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:46.912124  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:47.041978  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:47.210404 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:47.221343  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:47.620486  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:48.112658  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:48.161954 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:48.368113 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:48.368877  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:48.454854  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:48.579671  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:48.627626  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:48.640661  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:48.841157 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:16:48.935016 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:48.974195  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:16:49.026369  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:49.043543 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:49.292286 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:49.441058  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:49.514168  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:49.535418  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:49.568835  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:50.193935  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:50.701382 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:50.779168  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:50.878598 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:50.879442 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:50.945961  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:50.964308  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:50.979655  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:51.060056  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:51.060979  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:51.185837  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:51.464362 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:51.575405 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:51.707235 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:51.793627  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:52.160126  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:52.176272  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:52.184114  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:52.555411  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:53.026505  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:53.225570 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:53.227898  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:53.246946  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:53.455876  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:53.459435 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:53.532471  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:53.768805  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:53.789229 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:53.840294 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:54.029057  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:54.194306 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 02:16:54.195125 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:54.348131  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:16:54.596300  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:16:54.597245  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:54.630770  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:54.895968  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:54.980605  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:55.440447  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:55.489905  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:55.862830  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:55.877929  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:55.914283 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:55.914320 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:55.915315 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:55.916097  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:55.985443 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:56.204964  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:16:56.205708  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:56.389598 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:56.493588  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:56.785552 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:57.039901  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:57.370496  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:57.464777  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:57.586984  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:57.825606  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:58.114197  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:58.369060  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:58.385941  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:58.493969  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:58.512382 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:58.523216 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:58.732098  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:58.737694 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:58.856913 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:59.109027  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:59.242570 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:16:59.281807 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:59.571964  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:16:59.612838  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:59.756326  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:17:00.055056  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:17:00.058105  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:17:00.172595  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:00.210556  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:00.777105  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:00.895112  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:01.009665  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:01.071836 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:01.072785 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:01.194808 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:01.229931 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:01.236634  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:01.303961  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:01.304930  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:01.454640 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:01.703712  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:01.859869 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:02.223860  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:02.603857  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:02.614970  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:02.830053  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:02.900830  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:03.125710  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:03.144701  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:03.339414  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:03.631208 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:03.647116  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:03.683552 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:03.696910  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:03.766229 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:03.986059 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:04.172698  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:04.181293 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:04.280718 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:17:04.672130  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:17:04.747756  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:04.789480  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:17:04.900171  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:05.087998  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:05.191004  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:05.209995  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:05.618822  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:05.687168  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:05.819455  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:05.846649  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:06.013012 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:06.268708  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:06.414386 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:17:06.415340 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:06.577263 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:06.578191 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:06.604293  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240503 02:17:06.691313  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:06.961388 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:07.087991  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:07.304700  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:07.658833  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:07.827970  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:07.857980  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:08.044318  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:08.224237  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:08.276819  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:08.309986  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:08.471697 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:08.637725  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:08.813244 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:08.979590 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:09.028851 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:09.088989  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:09.392385 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:17:09.393237 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:09.482431  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:09.881331  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:09.916805  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:17:10.025615  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:17:10.026489  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:10.088881  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:10.434006  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:10.520853  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:10.575539  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:10.767091  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:10.924943 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:10.979135  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:11.132761  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:11.301746 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:11.360298 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:11.482470 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:17:11.489188 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:11.661870 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:11.691555  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:11.738170  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:12.102680  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:12.153425  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:12.502976  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:12.570147  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:12.897141  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:13.015918  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:13.190943  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:13.324342  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:13.444320 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:13.588014  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:13.603813  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:13.715509 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:13.841820 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:13.988126  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:14.036348 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:14.098701 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:14.282406  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:14.444669 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:17:14.494050  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:15.025849  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:17:15.026669  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:15.034358  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:15.248644  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:17:15.400336  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:15.520220  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:15.543565  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:15.778695  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:16.009519 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:16.131489  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:16.209295  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:16.293336 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:16.422358 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:16.434757 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:16.716316  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:16.744742 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:16.857092 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:17:16.981520  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:17:16.991214  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:17.269356  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:17.318092  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:17.790531  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:17.941534  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:18.066512  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:18.205622  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:18.300364  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:18.404525 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:18.597388  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:18.682279 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:18.844274  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:18.945356 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:19.113277 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:19.164790  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:19.373942 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:19.396879  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:19.477308 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:17:19.575088  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:19.966428  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:20.017290  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:20.302321  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:20.344326  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:17:20.486148  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:20.562345  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:20.735402  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:17:20.838817  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:20.928491 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:20.993885 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:21.120808  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:21.143435  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:21.411931 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:21.530046  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:21.693372 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:21.836887  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:21.837275 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:21.909844 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:22.141196  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:22.323873  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:17:22.613006  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:22.744089  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:22.802208  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:22.896054  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:23.121465  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:23.393839 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:23.512744  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:23.541383  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:23.648301  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:23.680104 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:23.696352 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:23.989862  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:24.250311 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:24.394502  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:24.469022 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:24.518244  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:24.588987 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:17:25.045496  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:25.103960  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:25.347641  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:17:25.479950  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:25.558738  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:25.631764  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:25.835353 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:26.070853 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:26.094410  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:26.187861 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:26.221108  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:17:26.222088  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:26.463758  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:26.504079  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:26.671846 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:26.723773 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:26.918781 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:27.090574  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:27.111141  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:27.571302  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:27.582479  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:27.624954  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:27.994781  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:28.014418  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:28.089615  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:28.384495 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:28.486698  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:28.508731 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:28.509006  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:28.640085 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:29.039528  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:29.040143  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:29.234788 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:29.275717 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:29.610378 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:17:29.645603  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:29.744192  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:30.059041  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:30.228530  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:30.468760  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:17:30.469653  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:30.835561 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:30.850824  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:30.885344  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:30.961692  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:31.070725  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:31.212289 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:31.259357 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:31.338172  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:31.593083  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:17:31.609604 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:31.633368 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:31.735672  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:31.770852  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:32.071250 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:32.085296  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:32.178500  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:32.699872  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:32.722129  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:17:32.938102  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:33.205992  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:33.208191  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:33.212090 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:33.294728  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:33.570065 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:33.640568  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:33.678781  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:33.700459 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:33.949599 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:34.208657  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:34.344826 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:34.401095  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:34.430277  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:34.690258  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:34.721011 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:17:35.238338  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:35.257967  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:35.402690  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:35.469990  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:17:35.687637 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:35.832345  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:35.915596 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:35.956240  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:35.965642  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:36.132619 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:36.371604  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:36.510459 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:36.831828  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:36.998077 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:37.048480  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:17:37.128337  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:37.154449  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:37.386178  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:37.456036 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:17:37.643723  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:37.848019  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:17:37.849010  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:37.982532  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:38.104406 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:38.351472  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:38.388782  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:38.429744 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:38.484975  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:38.597963 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:38.884604 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:38.922773  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:39.394713  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:39.620445 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:39.627097  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:39.799942 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:17:39.862533  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:39.944478  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:40.328130  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:40.534143  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:17:40.535044  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:40.627223  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:40.859792 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:40.876650 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:40.882299  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:40.924607 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:40.968792  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:41.202025  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:41.580036  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:41.699579 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:41.975737  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:42.077541 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:42.192585  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:17:42.202394  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:42.462801  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:42.527004 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:17:42.596716  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:42.895231  7817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:17:42.896261  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 80b4e6aa20f64473be8bcbf47baf66e4: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:42.919915  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:43.105489  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:43.113808  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:43.141069  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:43.219105 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:43.272974  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:43.378687 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:43.653355 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:43.672897  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:43.881299  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:44.165136 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:44.185249  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:44.391026  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:44.520314 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:44.787112  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:44.924129 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:17:45.127416  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:45.177256  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:45.284878  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:45.630316  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:17:45.631336  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:45.675022 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:45.698244  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:45.797174  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:45.805841 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:46.204146 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:46.220307  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:46.413688  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:46.527504 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:46.738555  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:46.825714 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:46.917090  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:47.234284  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:47.585386  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:17:47.621366  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:47.752234 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:17:47.758350  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:47.926811  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:48.082990  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:48.158090  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:48.327322 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:48.350793 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:48.368649  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:48.378407  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:48.834302 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:48.980033  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:49.111186  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:49.218425 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:49.446913 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:49.472687  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:49.597124  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:49.917740  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:50.053216  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:50.101619  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:50.258011 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:17:50.304934  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:50.456874  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:50.643615  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:17:50.770972 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:50.855937  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:50.880050 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:51.118000  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:51.358683  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:51.365739 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:51.640993  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:51.852778  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:51.911904 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:51.919698 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:52.061060  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:52.328104  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:52.444396  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:52.516620  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:52.588117  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:52.659986  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:17:52.866501 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:52.919582  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:53.097015 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:53.234550  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:17:53.246876 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:53.477013  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:53.726320 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:53.810133  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:53.938455  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:54.278957  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:54.393702 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:54.470681  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:54.631906 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:54.858860  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:54.928496  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:55.010046  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:55.031483  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:55.062206  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:55.275940  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:55.426122 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:17:55.609701 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:55.656314  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:17:55.657204  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:55.834092 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:55.902040  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:56.114986  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:56.364490 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:56.668257  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:56.671600  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:56.900875 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:56.972061 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:57.217777  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:57.356534  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:57.502521  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:57.513551  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:57.576535  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:57.701460  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:57.905678 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:58.134275  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:17:58.138537 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:58.294646  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:58.347187  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:17:58.348222  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:58.421422 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:58.867959  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:58.970140 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:59.009289  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:59.378880 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:59.441254  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:59.515328 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:59.817971  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:17:59.878011  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:18:00.038914  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:00.058748  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:18:00.129868  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:18:00.271034  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:00.637015  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:00.666355 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:18:00.667254 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:00.675328  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240503 02:18:00.788683  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:00.926007 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:01.298859 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:01.353348  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:01.550540  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:01.762138  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:02.005061 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:02.142315  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:02.280449  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:02.367982 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:02.372691  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:02.710768  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:02.756233  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:02.802419  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:02.929649 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:18:02.991616 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:03.169155  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:03.256902  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:03.315644 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:03.732872  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:03.759668  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:18:03.827455  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:03.890777 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:04.177834  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:04.295966  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:04.457810 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:04.787082  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:04.830852  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:04.909731 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:05.014626  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:05.307072  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:05.320140  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:05.416227  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:05.613972  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:05.757040  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:18:05.780344 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:18:05.781163 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:05.921440  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:06.117290 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:06.531404 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:06.532581  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:06.556600  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:07.113682  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:07.192653 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:07.288861  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:07.554869  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:07.612529 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:07.649086  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:07.809923  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:07.854022  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:07.944324 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:18:07.949201  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:08.026436  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:08.222097 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:08.505076  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:08.516350 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:08.743942 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:08.881618  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:18:09.036597  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:09.176311  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:09.177474  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:09.584812 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:09.605482  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:09.905644  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:10.189221  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:10.255827 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:10.325629  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:10.354509  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:10.379739  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:10.472093  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:10.618424 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:10.696151  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:10.811344  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:18:10.861609 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:10.901990 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:18:11.175485  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:11.373034 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:11.628746  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:11.763546  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:11.817392 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:12.276634  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:12.436380  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:12.666688  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:12.743981  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:12.808396 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:12.875489  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:12.951128  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:13.133105  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:13.189661 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:13.199057  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:13.227941 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:18:13.361770 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:13.687350  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:13.892560  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:13.892958 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:13.942819  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:18:14.167039  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:14.402340 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:14.402336  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:14.642472  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:14.684305  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:14.908972  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:15.182047  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:15.258445  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:15.381067 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:15.524065 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:15.573235  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:15.673077  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:15.787005  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:15.897897  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:18:16.043052 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:16.135720 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:18:16.406132  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:16.435366 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:16.464233  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:16.863971  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:16.930871 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:17.368752  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:17.465111  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:17.585167  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:17.611231  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:17.685549 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:17.949113  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:17.989568  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:18.036068 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:18.293309  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:18.348176 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:18:18.350111 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:18.562474  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:18.879942  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:19.026786 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:19.098882  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:19.196408  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:18:19.358449 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:19.447180  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:18:19.480312  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:19.845587 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:19.978529  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:20.138008  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:20.202399  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:20.214432  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:20.431525  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:20.442742  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:20.558930 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:20.644033 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:20.680608  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:20.905592  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240503 02:18:20.973453  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:21.071452  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:21.365374  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:21.536700 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:18:21.537794 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:21.763978  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:21.974923 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:22.366894  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:22.432933 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:22.439321  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:22.673934  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:22.741609  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:22.848330  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:22.955543  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:22.999100  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:23.156935 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:23.202934 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:23.234083  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:23.697743 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:18:23.752820  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:23.758919  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:24.291376 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:24.343729  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:24.519942 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:24.565785  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:18:24.811612  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:24.866153  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:25.078190 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:25.127338  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:25.156034  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:25.227952  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:25.288630  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:25.435084  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:25.488853  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:25.585532 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:25.732210  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:25.901656 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:25.968478  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:18:26.341415  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:26.407186  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:26.523068 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:26.731381  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:26.795856 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:18:26.925453 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:27.227313 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:27.293561  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:27.432389  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:27.478507  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:27.677699 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:27.730535  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:27.803038  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:27.817221  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:27.892741  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:28.385069 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:28.386930  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:28.781028  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:28.982208  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:29.066819 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:29.086534 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240503 02:18:29.180943  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:29.447549 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:29.574790  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:29.619597 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:29.619982  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:18:29.878296  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:18:29.891520  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:29.972177 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:30.037389  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:30.110412  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:30.146664  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:30.323947  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:30.435660  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:30.570413  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:30.697062 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:31.008586  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:18:31.096675  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:31.452281  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:31.558429 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:31.644201  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:31.970410 10197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:18:31.971779 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:32.212194  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:32.346297 10064 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:32.370251  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:32.423794 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:32.424654  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:32.664211  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:32.674423  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:32.817104  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:32.997352  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:33.074441 10197 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:33.303236  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:33.765558  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
W20240503 02:18:33.954102 10197 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:18:34.148900 25663 consensus_queue.cc:579] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Leader has been unable to successfully communicate with peer 03f99420df704238ab300ad6e58d878c for more than 300 seconds (300.131s)
W20240503 02:18:34.151002  8168 consensus_peers.cc:462] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:18:34.158108  8232 consensus_queue.cc:237] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:18:34.160569 10126 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:18:34.160743  7968 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:18:34.161255 27041 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:18:34.163477 27041 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:18:34.165920 27042 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:34.168994  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ecb5c78a22a74728ab40f1da8f90388c with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:34.170826  7823 catalog_manager.cc:5484] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 7 to 8, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:34.171885  7968 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:34.180789 10126 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:34.205606  8232 consensus_queue.cc:237] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
W20240503 02:18:34.206364  7817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20240503 02:18:34.208693 10126 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:18:34.208693  7968 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 02:18:34.209161  8167 consensus_peers.cc:485] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecb5c78a22a74728ab40f1da8f90388c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:34.211267 25663 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:18:34.211979 27041 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:18:34.216740 27041 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:18:34.219601  7968 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:18:34.220372 10126 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
W20240503 02:18:34.221753  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ecb5c78a22a74728ab40f1da8f90388c: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:18:34.224802  7814 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ecb5c78a22a74728ab40f1da8f90388c with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:34.226305  7826 catalog_manager.cc:5484] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 8 to 9, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
W20240503 02:18:34.238842  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:18:34.303488 25663 consensus_queue.cc:579] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Leader has been unable to successfully communicate with peer 03f99420df704238ab300ad6e58d878c for more than 300 seconds (300.163s)
I20240503 02:18:34.308601  8232 consensus_queue.cc:237] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:34.311537  9995 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:18:34.312026 26855 consensus_queue.cc:579] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Leader has been unable to successfully communicate with peer 03f99420df704238ab300ad6e58d878c for more than 300 seconds (300.314s)
I20240503 02:18:34.312084 27041 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 02:18:34.312544  8165 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32e527fa9b2f441cb5668435681b4ff6. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:34.313015  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:34.317412 27041 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:18:34.322386  9995 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
W20240503 02:18:34.323067 10064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 02:18:34.335048 10126 consensus_queue.cc:237] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
W20240503 02:18:34.336652 10197 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:18:34.337294  7814 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 32e527fa9b2f441cb5668435681b4ff6 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:34.341358  7827 catalog_manager.cc:5484] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 9 to 10, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:34.348380  8361 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:18:34.348513  8100 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:18:34.349345 26855 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:18:34.350929 27109 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:18:34.366869 26855 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:34.367673  8362 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:34.369458  8100 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:34.371533  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:34.373205  7823 catalog_manager.cc:5484] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 8 to 10, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 2 leader_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:34.375725 27065 consensus_queue.cc:579] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Leader has been unable to successfully communicate with peer 03f99420df704238ab300ad6e58d878c for more than 300 seconds (300.360s)
I20240503 02:18:34.383304 10126 consensus_queue.cc:237] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } }
I20240503 02:18:34.385468  8100 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:18:34.385550 10261 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 13, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:18:34.386078 27109 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:18:34.387033  8362 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:18:34.387614 27109 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:18:34.388849 27107 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } } }
I20240503 02:18:34.390281  8100 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } } }
I20240503 02:18:34.392395  7814 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 02:18:34.394068 10061 consensus_peers.cc:485] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): Couldn't send request to peer 2942efc0ecb448d2ba2f6bca20fbb29c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e4acc8b5ceb4cd5b0f3708ff4f19298. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:34.396185  7827 catalog_manager.cc:5484] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: config changed from index 10 to 11, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New cstate: current_term: 2 leader_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:34.396306  8362 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } } }
I20240503 02:18:34.400384  8100 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240503 02:18:34.400781  9995 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240503 02:18:34.401015 25993 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 02:18:34.401638 25993 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 02:18:34.410183 27065 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 13 to 15, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:34.411067  8100 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 13 to 15, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:34.413194  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f44b2b53de84a87a8757c0d00bfed37 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:34.415480  7827 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 13 to 15, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 2 leader_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:34.417529  9995 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 13 to 15, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:34.425122 10261 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 13, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } }
I20240503 02:18:34.427826  9995 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20240503 02:18:34.428333 27065 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240503 02:18:34.428699 10194 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): Couldn't send request to peer 2942efc0ecb448d2ba2f6bca20fbb29c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f44b2b53de84a87a8757c0d00bfed37. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:34.429503  8100 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20240503 02:18:34.430087 27117 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 02:18:34.430668 25993 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } } }
I20240503 02:18:34.431452  9995 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } } }
I20240503 02:18:34.434257  7816 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0f44b2b53de84a87a8757c0d00bfed37 with cas_config_opid_index 15: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:34.436041  7827 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 15 to 16, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New cstate: current_term: 2 leader_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:34.436403  8100 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } } }
I20240503 02:18:34.474336 27041 consensus_queue.cc:579] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Leader has been unable to successfully communicate with peer 03f99420df704238ab300ad6e58d878c for more than 300 seconds (300.297s)
I20240503 02:18:34.479393  8232 consensus_queue.cc:237] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
W20240503 02:18:34.484637  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397): Couldn't send request to peer b85e85e631ab4e04a86bfe5a9f0aa512. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ceae1c7c45a34b24badbbe23cb9e0176. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:34.485441  8100 raft_consensus.cc:1261] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:18:34.485704  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:34.486017 27041 consensus_queue.cc:1035] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:18:34.492138 25663 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:18:34.495388  7814 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ceae1c7c45a34b24badbbe23cb9e0176 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:34.495843  8100 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:18:34.501883  7823 catalog_manager.cc:5484] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 6 to 7, NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } } }
I20240503 02:18:34.600396 25786 consensus_queue.cc:579] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Leader has been unable to successfully communicate with peer 03f99420df704238ab300ad6e58d878c for more than 300 seconds (300.140s)
I20240503 02:18:34.609552  8100 consensus_queue.cc:237] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } }
W20240503 02:18:34.612432  8035 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): Couldn't send request to peer 2942efc0ecb448d2ba2f6bca20fbb29c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:34.612457 10261 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:18:34.612967 27153 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 02:18:34.613310  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:34.615700 25786 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } } }
I20240503 02:18:34.616305 10261 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } } }
I20240503 02:18:34.619958  7816 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 007b59de142c4b95966494ba097b1a09 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:34.620640  7827 catalog_manager.cc:5484] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 9 to 10, NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) added. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:18:34.678555  9932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 02:18:34.691464 27158 ts_tablet_manager.cc:902] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:18:34.692577 27158 tablet_copy_client.cc:308] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:18:34.693334  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet ecb5c78a22a74728ab40f1da8f90388c from peer ca05c9a8bee148dab8a2a982b8fa2ea4 ({username='slave'} at 127.0.145.8:44853)
I20240503 02:18:34.693470  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet ecb5c78a22a74728ab40f1da8f90388c from peer ca05c9a8bee148dab8a2a982b8fa2ea4 at {username='slave'} at 127.0.145.8:44853: session id = ca05c9a8bee148dab8a2a982b8fa2ea4-ecb5c78a22a74728ab40f1da8f90388c
I20240503 02:18:34.695888  8252 tablet_copy_source_session.cc:205] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:18:34.697388 27158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecb5c78a22a74728ab40f1da8f90388c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:34.701850 27158 tablet_copy_client.cc:783] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 5 data blocks...
I20240503 02:18:34.708417 27158 tablet_copy_client.cc:647] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:34.710211 27158 tablet_copy_client.cc:523] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:18:34.713721 27158 tablet_bootstrap.cc:492] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap starting.
I20240503 02:18:34.721571 27168 ts_tablet_manager.cc:902] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:18:34.722900 27168 tablet_copy_client.cc:308] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:18:34.723624  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet 32e527fa9b2f441cb5668435681b4ff6 from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:44937)
I20240503 02:18:34.723783  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet 32e527fa9b2f441cb5668435681b4ff6 from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:44937: session id = d826d4dae2aa4e05b2f64c9b17a1126f-32e527fa9b2f441cb5668435681b4ff6
I20240503 02:18:34.726862  8252 tablet_copy_source_session.cc:205] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:18:34.728598 27168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e527fa9b2f441cb5668435681b4ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:34.731729 27158 tablet_bootstrap.cc:492] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:34.732118 27158 tablet_bootstrap.cc:492] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4: Bootstrap complete.
I20240503 02:18:34.732403 27158 ts_tablet_manager.cc:1367] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent bootstrapping tablet: real 0.019s	user 0.018s	sys 0.000s
I20240503 02:18:34.733036 27158 raft_consensus.cc:348] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:18:34.733240 27158 raft_consensus.cc:729] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca05c9a8bee148dab8a2a982b8fa2ea4, State: Initialized, Role: LEARNER
I20240503 02:18:34.733434 27158 consensus_queue.cc:260] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:18:34.734047 27158 ts_tablet_manager.cc:1398] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:18:34.735211  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-ecb5c78a22a74728ab40f1da8f90388c received from {username='slave'} at 127.0.145.8:44853
I20240503 02:18:34.735373  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session ca05c9a8bee148dab8a2a982b8fa2ea4-ecb5c78a22a74728ab40f1da8f90388c on tablet ecb5c78a22a74728ab40f1da8f90388c with peer ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:18:34.737098 27168 tablet_copy_client.cc:783] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 5 data blocks...
I20240503 02:18:34.746090 27168 tablet_copy_client.cc:647] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:34.748404 27168 tablet_copy_client.cc:523] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:18:34.752249 27168 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
I20240503 02:18:34.762727 10307 maintenance_manager.cc:419] P ca05c9a8bee148dab8a2a982b8fa2ea4: Scheduling UndoDeltaBlockGCOp(ecb5c78a22a74728ab40f1da8f90388c): 4923 bytes on disk
I20240503 02:18:34.763289 10198 maintenance_manager.cc:643] P ca05c9a8bee148dab8a2a982b8fa2ea4: UndoDeltaBlockGCOp(ecb5c78a22a74728ab40f1da8f90388c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240503 02:18:34.774175 27168 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:34.774662 27168 tablet_bootstrap.cc:492] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:18:34.775007 27168 ts_tablet_manager.cc:1367] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 0.023s	user 0.014s	sys 0.008s
I20240503 02:18:34.775749 27168 raft_consensus.cc:348] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:34.776023 27168 raft_consensus.cc:729] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:18:34.776230 27168 consensus_queue.cc:260] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:34.776922 27168 ts_tablet_manager.cc:1398] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:18:34.778532  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-32e527fa9b2f441cb5668435681b4ff6 received from {username='slave'} at 127.0.145.7:44937
I20240503 02:18:34.778663  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-32e527fa9b2f441cb5668435681b4ff6 on tablet 32e527fa9b2f441cb5668435681b4ff6 with peer d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:18:34.856746 27200 ts_tablet_manager.cc:902] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c: Initiating tablet copy from peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
I20240503 02:18:34.858000 27200 tablet_copy_client.cc:308] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.7:39415
I20240503 02:18:34.858892 10148 tablet_copy_service.cc:140] P d826d4dae2aa4e05b2f64c9b17a1126f: Received BeginTabletCopySession request for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 from peer 2942efc0ecb448d2ba2f6bca20fbb29c ({username='slave'} at 127.0.145.3:44249)
I20240503 02:18:34.859064 10148 tablet_copy_service.cc:161] P d826d4dae2aa4e05b2f64c9b17a1126f: Beginning new tablet copy session on tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 from peer 2942efc0ecb448d2ba2f6bca20fbb29c at {username='slave'} at 127.0.145.3:44249: session id = 2942efc0ecb448d2ba2f6bca20fbb29c-7e4acc8b5ceb4cd5b0f3708ff4f19298
I20240503 02:18:34.861871 10148 tablet_copy_source_session.cc:205] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:18:34.863512 27200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:34.868037 27200 tablet_copy_client.cc:783] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Starting download of 5 data blocks...
I20240503 02:18:34.871812 27205 ts_tablet_manager.cc:902] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c: Initiating tablet copy from peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
I20240503 02:18:34.874219 27205 tablet_copy_client.cc:308] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.8:38885
I20240503 02:18:34.875030 10281 tablet_copy_service.cc:140] P ca05c9a8bee148dab8a2a982b8fa2ea4: Received BeginTabletCopySession request for tablet 0f44b2b53de84a87a8757c0d00bfed37 from peer 2942efc0ecb448d2ba2f6bca20fbb29c ({username='slave'} at 127.0.145.3:58371)
I20240503 02:18:34.875237 10281 tablet_copy_service.cc:161] P ca05c9a8bee148dab8a2a982b8fa2ea4: Beginning new tablet copy session on tablet 0f44b2b53de84a87a8757c0d00bfed37 from peer 2942efc0ecb448d2ba2f6bca20fbb29c at {username='slave'} at 127.0.145.3:58371: session id = 2942efc0ecb448d2ba2f6bca20fbb29c-0f44b2b53de84a87a8757c0d00bfed37
I20240503 02:18:34.875494 27200 tablet_copy_client.cc:647] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:34.877230 27200 tablet_copy_client.cc:523] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:18:34.878664 10281 tablet_copy_source_session.cc:205] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:18:34.880507 27200 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:18:34.886399 27205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f44b2b53de84a87a8757c0d00bfed37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:34.891678 27205 tablet_copy_client.cc:783] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Starting download of 5 data blocks...
I20240503 02:18:34.897766 27200 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:34.898286 27205 tablet_copy_client.cc:647] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:34.898336 27200 tablet_bootstrap.cc:492] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap complete.
I20240503 02:18:34.898689 27200 ts_tablet_manager.cc:1367] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.018s	user 0.020s	sys 0.000s
I20240503 02:18:34.899456 27200 raft_consensus.cc:348] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } }
I20240503 02:18:34.899730 27200 raft_consensus.cc:729] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: LEARNER
I20240503 02:18:34.899953 27200 consensus_queue.cc:260] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } }
I20240503 02:18:34.900547 27200 ts_tablet_manager.cc:1398] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:18:34.901297 10148 tablet_copy_service.cc:342] P d826d4dae2aa4e05b2f64c9b17a1126f: Request end of tablet copy session 2942efc0ecb448d2ba2f6bca20fbb29c-7e4acc8b5ceb4cd5b0f3708ff4f19298 received from {username='slave'} at 127.0.145.3:44249
I20240503 02:18:34.901449 10148 tablet_copy_service.cc:433] P d826d4dae2aa4e05b2f64c9b17a1126f: ending tablet copy session 2942efc0ecb448d2ba2f6bca20fbb29c-7e4acc8b5ceb4cd5b0f3708ff4f19298 on tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 with peer 2942efc0ecb448d2ba2f6bca20fbb29c
I20240503 02:18:34.902258 27205 tablet_copy_client.cc:523] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:18:34.906059 27205 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
I20240503 02:18:34.924105 27205 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:34.924489 27205 tablet_bootstrap.cc:492] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap complete.
I20240503 02:18:34.924738 27205 ts_tablet_manager.cc:1367] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.019s	user 0.016s	sys 0.003s
I20240503 02:18:34.925355 27205 raft_consensus.cc:348] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } }
I20240503 02:18:34.925577 27205 raft_consensus.cc:729] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: LEARNER
I20240503 02:18:34.925762 27205 consensus_queue.cc:260] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } }
I20240503 02:18:34.926296 27205 ts_tablet_manager.cc:1398] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:18:34.927297 10281 tablet_copy_service.cc:342] P ca05c9a8bee148dab8a2a982b8fa2ea4: Request end of tablet copy session 2942efc0ecb448d2ba2f6bca20fbb29c-0f44b2b53de84a87a8757c0d00bfed37 received from {username='slave'} at 127.0.145.3:58371
I20240503 02:18:34.927448 10281 tablet_copy_service.cc:433] P ca05c9a8bee148dab8a2a982b8fa2ea4: ending tablet copy session 2942efc0ecb448d2ba2f6bca20fbb29c-0f44b2b53de84a87a8757c0d00bfed37 on tablet 0f44b2b53de84a87a8757c0d00bfed37 with peer 2942efc0ecb448d2ba2f6bca20fbb29c
I20240503 02:18:34.962884 27065 consensus_queue.cc:579] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Leader has been unable to successfully communicate with peer 03f99420df704238ab300ad6e58d878c for more than 300 seconds (300.205s)
I20240503 02:18:34.966969 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling UndoDeltaBlockGCOp(32e527fa9b2f441cb5668435681b4ff6): 4923 bytes on disk
I20240503 02:18:34.967705 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: UndoDeltaBlockGCOp(32e527fa9b2f441cb5668435681b4ff6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
I20240503 02:18:34.973369 10261 consensus_queue.cc:237] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:18:34.976189  8100 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:18:34.976928 27065 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:18:34.977021  8362 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:18:34.977766 27065 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:18:34.984247 27065 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
W20240503 02:18:34.984973 10064 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:18:34.987028  8100 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:34.994539  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b64f66ace86e42f3ac213ba85bcab584 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:34.994370  7823 catalog_manager.cc:5484] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 9 to 11, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 2 leader_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:34.996574  8362 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:35.001396 10261 consensus_queue.cc:237] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:35.005240  8100 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:18:35.006441 27117 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:18:35.012624  8362 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240503 02:18:35.013129 10194 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b64f66ace86e42f3ac213ba85bcab584. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:35.011886 27065 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:18:35.013234 27117 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:18:35.013777  8100 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:18:35.018363  8361 raft_consensus.cc:2932] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:18:35.021178  7823 catalog_manager.cc:5484] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 11 to 12, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 2 leader_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:35.022863  7816 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b64f66ace86e42f3ac213ba85bcab584 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:35.038645  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling UndoDeltaBlockGCOp(7e4acc8b5ceb4cd5b0f3708ff4f19298): 4838 bytes on disk
I20240503 02:18:35.039242  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: UndoDeltaBlockGCOp(7e4acc8b5ceb4cd5b0f3708ff4f19298) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240503 02:18:35.040215  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling UndoDeltaBlockGCOp(0f44b2b53de84a87a8757c0d00bfed37): 4836 bytes on disk
I20240503 02:18:35.040664  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: UndoDeltaBlockGCOp(0f44b2b53de84a87a8757c0d00bfed37) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 02:18:35.066334 27246 ts_tablet_manager.cc:902] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:18:35.067582 27246 tablet_copy_client.cc:308] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:18:35.072777  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet ceae1c7c45a34b24badbbe23cb9e0176 from peer b85e85e631ab4e04a86bfe5a9f0aa512 ({username='slave'} at 127.0.145.6:54693)
I20240503 02:18:35.072937  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet ceae1c7c45a34b24badbbe23cb9e0176 from peer b85e85e631ab4e04a86bfe5a9f0aa512 at {username='slave'} at 127.0.145.6:54693: session id = b85e85e631ab4e04a86bfe5a9f0aa512-ceae1c7c45a34b24badbbe23cb9e0176
I20240503 02:18:35.075301  8252 tablet_copy_source_session.cc:205] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:18:35.076815 27246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceae1c7c45a34b24badbbe23cb9e0176. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:35.082049 27246 tablet_copy_client.cc:783] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 5 data blocks...
I20240503 02:18:35.088177 27246 tablet_copy_client.cc:647] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:35.089798 27246 tablet_copy_client.cc:523] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:18:35.091907 25145 consensus_queue.cc:579] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Leader has been unable to successfully communicate with peer 03f99420df704238ab300ad6e58d878c for more than 300 seconds (300.307s)
I20240503 02:18:35.093183 27246 tablet_bootstrap.cc:492] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap starting.
W20240503 02:18:35.094043  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:18:35.104681  9995 consensus_queue.cc:237] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:18:35.107895 10126 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 02:18:35.108003  8100 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 02:18:35.108816 27252 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:18:35.109714 27252 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:18:35.114532 25145 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:35.115480  8100 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:35.117659  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 72e2bb75da654e8ab45bba8ac077f28c with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:35.119894  7823 catalog_manager.cc:5484] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 12 to 14, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 2 leader_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:35.127070  9995 consensus_queue.cc:237] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } }
W20240503 02:18:35.129559  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): Couldn't send request to peer 8d7c037d50c241b484ee843b06313a64. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:35.133144 27246 tablet_bootstrap.cc:492] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:35.133510 27246 tablet_bootstrap.cc:492] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512: Bootstrap complete.
I20240503 02:18:35.133749 27246 ts_tablet_manager.cc:1367] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent bootstrapping tablet: real 0.041s	user 0.017s	sys 0.007s
I20240503 02:18:35.134385 27246 raft_consensus.cc:348] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:18:35.134568 27246 raft_consensus.cc:729] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Initialized, Role: LEARNER
I20240503 02:18:35.134770 27246 consensus_queue.cc:260] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: true } }
I20240503 02:18:35.135277 27246 ts_tablet_manager.cc:1398] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240503 02:18:35.135975  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-ceae1c7c45a34b24badbbe23cb9e0176 received from {username='slave'} at 127.0.145.6:54693
I20240503 02:18:35.136098  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session b85e85e631ab4e04a86bfe5a9f0aa512-ceae1c7c45a34b24badbbe23cb9e0176 on tablet ceae1c7c45a34b24badbbe23cb9e0176 with peer b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:18:35.137073 10127 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240503 02:18:35.137627  8100 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
W20240503 02:18:35.137941  8038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:18:35.137952 27252 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:18:35.140633 10126 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:35.142060 27205 ts_tablet_manager.cc:902] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: Initiating tablet copy from peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309)
I20240503 02:18:35.154068 25145 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:18:35.158176  7817 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 72e2bb75da654e8ab45bba8ac077f28c with cas_config_opid_index 14: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:35.158430 10126 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:18:35.159029  7823 catalog_manager.cc:5484] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 14 to 15, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) added. New cstate: current_term: 2 leader_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:35.169165 25145 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 02:18:35.170922 27205 tablet_copy_client.cc:308] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35309
I20240503 02:18:35.171519  8100 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } } }
I20240503 02:18:35.174810  8122 tablet_copy_service.cc:140] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Received BeginTabletCopySession request for tablet 007b59de142c4b95966494ba097b1a09 from peer 2942efc0ecb448d2ba2f6bca20fbb29c ({username='slave'} at 127.0.145.3:57873)
I20240503 02:18:35.174971  8122 tablet_copy_service.cc:161] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Beginning new tablet copy session on tablet 007b59de142c4b95966494ba097b1a09 from peer 2942efc0ecb448d2ba2f6bca20fbb29c at {username='slave'} at 127.0.145.3:57873: session id = 2942efc0ecb448d2ba2f6bca20fbb29c-007b59de142c4b95966494ba097b1a09
I20240503 02:18:35.178849  8122 tablet_copy_source_session.cc:205] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:18:35.180186 27205 ts_tablet_manager.cc:1875] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:18:35.187682 27205 ts_tablet_manager.cc:1888] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:18:35.188079 27205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 007b59de142c4b95966494ba097b1a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:35.196745 10261 raft_consensus.cc:1203] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240503 02:18:35.200619 27205 tablet_copy_client.cc:783] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Starting download of 5 data blocks...
I20240503 02:18:35.213661 10126 raft_consensus.cc:1203] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240503 02:18:35.218855 27205 tablet_copy_client.cc:647] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:35.220831 27205 tablet_copy_client.cc:523] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:18:35.223997 27205 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap starting.
W20240503 02:18:35.227610  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:18:35.241171 27205 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:35.241650 27205 tablet_bootstrap.cc:492] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: Bootstrap complete.
I20240503 02:18:35.241931 27205 ts_tablet_manager.cc:1367] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent bootstrapping tablet: real 0.018s	user 0.012s	sys 0.003s
I20240503 02:18:35.242686 27205 raft_consensus.cc:348] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } }
I20240503 02:18:35.242941 27205 raft_consensus.cc:729] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Initialized, Role: LEARNER
I20240503 02:18:35.243149 27205 consensus_queue.cc:260] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: true } }
I20240503 02:18:35.243757 27205 ts_tablet_manager.cc:1398] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 02:18:35.244448  8122 tablet_copy_service.cc:342] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Request end of tablet copy session 2942efc0ecb448d2ba2f6bca20fbb29c-007b59de142c4b95966494ba097b1a09 received from {username='slave'} at 127.0.145.3:57873
I20240503 02:18:35.244583  8122 tablet_copy_service.cc:433] P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: ending tablet copy session 2942efc0ecb448d2ba2f6bca20fbb29c-007b59de142c4b95966494ba097b1a09 on tablet 007b59de142c4b95966494ba097b1a09 with peer 2942efc0ecb448d2ba2f6bca20fbb29c
W20240503 02:18:35.290800  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 72e2bb75da654e8ab45bba8ac077f28c: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:18:35.292620  8278 maintenance_manager.cc:419] P 2942efc0ecb448d2ba2f6bca20fbb29c: Scheduling UndoDeltaBlockGCOp(007b59de142c4b95966494ba097b1a09): 4921 bytes on disk
I20240503 02:18:35.293246  8169 maintenance_manager.cc:643] P 2942efc0ecb448d2ba2f6bca20fbb29c: UndoDeltaBlockGCOp(007b59de142c4b95966494ba097b1a09) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240503 02:18:35.311045  8232 raft_consensus.cc:1203] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11]   Dedup: 2.11->[]
I20240503 02:18:35.362797 10042 maintenance_manager.cc:419] P b85e85e631ab4e04a86bfe5a9f0aa512: Scheduling UndoDeltaBlockGCOp(ceae1c7c45a34b24badbbe23cb9e0176): 4923 bytes on disk
I20240503 02:18:35.363471  9933 maintenance_manager.cc:643] P b85e85e631ab4e04a86bfe5a9f0aa512: UndoDeltaBlockGCOp(ceae1c7c45a34b24badbbe23cb9e0176) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 02:18:35.375481  8232 raft_consensus.cc:1203] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.15->[2.16-2.16]   Dedup: 2.16->[]
W20240503 02:18:35.500288  8168 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:18:35.530876 27331 ts_tablet_manager.cc:902] T b64f66ace86e42f3ac213ba85bcab584 P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
I20240503 02:18:35.532281 27331 tablet_copy_client.cc:308] T b64f66ace86e42f3ac213ba85bcab584 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.8:38885
I20240503 02:18:35.537554 10281 tablet_copy_service.cc:140] P ca05c9a8bee148dab8a2a982b8fa2ea4: Received BeginTabletCopySession request for tablet b64f66ace86e42f3ac213ba85bcab584 from peer d826d4dae2aa4e05b2f64c9b17a1126f ({username='slave'} at 127.0.145.7:51095)
I20240503 02:18:35.537709 10281 tablet_copy_service.cc:161] P ca05c9a8bee148dab8a2a982b8fa2ea4: Beginning new tablet copy session on tablet b64f66ace86e42f3ac213ba85bcab584 from peer d826d4dae2aa4e05b2f64c9b17a1126f at {username='slave'} at 127.0.145.7:51095: session id = d826d4dae2aa4e05b2f64c9b17a1126f-b64f66ace86e42f3ac213ba85bcab584
I20240503 02:18:35.539850 10281 tablet_copy_source_session.cc:205] T b64f66ace86e42f3ac213ba85bcab584 P ca05c9a8bee148dab8a2a982b8fa2ea4: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:18:35.541059 27331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64f66ace86e42f3ac213ba85bcab584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:35.546424 27331 tablet_copy_client.cc:783] T b64f66ace86e42f3ac213ba85bcab584 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 5 data blocks...
I20240503 02:18:35.552402 27331 tablet_copy_client.cc:647] T b64f66ace86e42f3ac213ba85bcab584 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:35.554150 27331 tablet_copy_client.cc:523] T b64f66ace86e42f3ac213ba85bcab584 P d826d4dae2aa4e05b2f64c9b17a1126f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:18:35.558035 27331 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap starting.
I20240503 02:18:35.561889  8232 raft_consensus.cc:1203] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240503 02:18:35.585122 27331 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:35.585474 27331 tablet_bootstrap.cc:492] T b64f66ace86e42f3ac213ba85bcab584 P d826d4dae2aa4e05b2f64c9b17a1126f: Bootstrap complete.
I20240503 02:18:35.585688 27331 ts_tablet_manager.cc:1367] T b64f66ace86e42f3ac213ba85bcab584 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent bootstrapping tablet: real 0.028s	user 0.025s	sys 0.000s
I20240503 02:18:35.586315 27331 raft_consensus.cc:348] T b64f66ace86e42f3ac213ba85bcab584 P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:35.586514 27331 raft_consensus.cc:729] T b64f66ace86e42f3ac213ba85bcab584 P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d826d4dae2aa4e05b2f64c9b17a1126f, State: Initialized, Role: LEARNER
I20240503 02:18:35.586735 27331 consensus_queue.cc:260] T b64f66ace86e42f3ac213ba85bcab584 P d826d4dae2aa4e05b2f64c9b17a1126f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:35.587195 27331 ts_tablet_manager.cc:1398] T b64f66ace86e42f3ac213ba85bcab584 P d826d4dae2aa4e05b2f64c9b17a1126f: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240503 02:18:35.588030 10281 tablet_copy_service.cc:342] P ca05c9a8bee148dab8a2a982b8fa2ea4: Request end of tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-b64f66ace86e42f3ac213ba85bcab584 received from {username='slave'} at 127.0.145.7:51095
I20240503 02:18:35.588181 10281 tablet_copy_service.cc:433] P ca05c9a8bee148dab8a2a982b8fa2ea4: ending tablet copy session d826d4dae2aa4e05b2f64c9b17a1126f-b64f66ace86e42f3ac213ba85bcab584 on tablet b64f66ace86e42f3ac213ba85bcab584 with peer d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:18:35.638413 27266 raft_consensus.cc:1050] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 to VOTER
I20240503 02:18:35.639292 27266 consensus_queue.cc:237] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:18:35.641652 27131 raft_consensus.cc:1050] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f to VOTER
I20240503 02:18:35.641636 10126 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:18:35.642103  7968 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:18:35.642593 27263 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:18:35.642413 27131 consensus_queue.cc:237] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:18:35.643164 10261 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:18:35.643726 27263 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:18:35.644795 27266 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:18:35.645931 10127 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:18:35.646723  9995 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:18:35.647297 27264 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:18:35.647516 27266 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:18:35.648138  8168 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:35.654938 27345 ts_tablet_manager.cc:902] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: Initiating tablet copy from peer b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397)
I20240503 02:18:35.654947 27264 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:35.659039 10261 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:35.662932  7969 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:35.671227  9994 raft_consensus.cc:1203] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:18:35.671805 27131 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:35.675837  9994 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:35.683236  7828 catalog_manager.cc:5484] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 10 to 11, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:35.683941 10126 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:35.687199  7827 catalog_manager.cc:5484] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 reported cstate change: config changed from index 9 to 10, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:35.690585 10127 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { replace: true } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:35.695874 27345 tablet_copy_client.cc:308] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.6:40397
I20240503 02:18:35.696851 10016 tablet_copy_service.cc:140] P b85e85e631ab4e04a86bfe5a9f0aa512: Received BeginTabletCopySession request for tablet 72e2bb75da654e8ab45bba8ac077f28c from peer 8d7c037d50c241b484ee843b06313a64 ({username='slave'} at 127.0.145.4:52991)
I20240503 02:18:35.697106 10016 tablet_copy_service.cc:161] P b85e85e631ab4e04a86bfe5a9f0aa512: Beginning new tablet copy session on tablet 72e2bb75da654e8ab45bba8ac077f28c from peer 8d7c037d50c241b484ee843b06313a64 at {username='slave'} at 127.0.145.4:52991: session id = 8d7c037d50c241b484ee843b06313a64-72e2bb75da654e8ab45bba8ac077f28c
I20240503 02:18:35.701685 10016 tablet_copy_source_session.cc:205] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:18:35.701900  8232 consensus_queue.cc:237] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:18:35.703649 27345 ts_tablet_manager.cc:1875] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:18:35.704305 10127 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:18:35.704895 27131 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:18:35.707515  8232 consensus_queue.cc:237] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:18:35.709486 10261 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:18:35.709964 10127 raft_consensus.cc:1261] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:18:35.709652 27131 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:35.710141 27345 ts_tablet_manager.cc:1888] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:18:35.710415 27345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e2bb75da654e8ab45bba8ac077f28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:35.710733 27266 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:18:35.711503  9994 raft_consensus.cc:1261] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:18:35.712109 27263 consensus_queue.cc:1035] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:18:35.712154 27265 consensus_queue.cc:1035] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:18:35.713317 10127 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:35.713757  9994 raft_consensus.cc:2932] T 32e527fa9b2f441cb5668435681b4ff6 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:35.714928 27345 tablet_copy_client.cc:783] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: tablet copy: Starting download of 5 data blocks...
I20240503 02:18:35.718854 27265 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:35.719231  7830 catalog_manager.cc:5484] T 32e527fa9b2f441cb5668435681b4ff6 P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: config changed from index 11 to 12, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:35.721458 10261 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:35.723177 10174 maintenance_manager.cc:419] P d826d4dae2aa4e05b2f64c9b17a1126f: Scheduling UndoDeltaBlockGCOp(b64f66ace86e42f3ac213ba85bcab584): 4923 bytes on disk
I20240503 02:18:35.723909 10065 maintenance_manager.cc:643] P d826d4dae2aa4e05b2f64c9b17a1126f: UndoDeltaBlockGCOp(b64f66ace86e42f3ac213ba85bcab584) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240503 02:18:35.729784  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ecb5c78a22a74728ab40f1da8f90388c with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:35.732414  7823 catalog_manager.cc:5484] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 10 to 11, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:35.733779  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 32e527fa9b2f441cb5668435681b4ff6 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:35.741158 10124 raft_consensus.cc:2932] T ecb5c78a22a74728ab40f1da8f90388c P d826d4dae2aa4e05b2f64c9b17a1126f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:35.742556 27345 tablet_copy_client.cc:647] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:35.744946 27345 tablet_copy_client.cc:523] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:18:35.750936 27345 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: Bootstrap starting.
I20240503 02:18:35.777621  7950 tablet_service.cc:1514] Processing DeleteTablet for tablet ecb5c78a22a74728ab40f1da8f90388c with delete_type TABLET_DATA_TOMBSTONED (TS d986fcfb926e47baa7d2d5b99f697334 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:54504
I20240503 02:18:35.787106 27371 tablet_replica.cc:330] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334: stopping tablet replica
I20240503 02:18:35.789280 27371 raft_consensus.cc:2229] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:18:35.789996 27371 raft_consensus.cc:2258] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:18:35.791338 27371 ts_tablet_manager.cc:1875] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:18:35.812460 27345 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:35.812978 27345 tablet_bootstrap.cc:492] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: Bootstrap complete.
I20240503 02:18:35.813412 27345 ts_tablet_manager.cc:1367] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: Time spent bootstrapping tablet: real 0.063s	user 0.031s	sys 0.017s
I20240503 02:18:35.814474 27345 raft_consensus.cc:348] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } }
I20240503 02:18:35.814956 27345 raft_consensus.cc:729] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Initialized, Role: LEARNER
I20240503 02:18:35.815268 27345 consensus_queue.cc:260] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: true } }
I20240503 02:18:35.816128 27345 ts_tablet_manager.cc:1398] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:18:35.816936 10016 tablet_copy_service.cc:342] P b85e85e631ab4e04a86bfe5a9f0aa512: Request end of tablet copy session 8d7c037d50c241b484ee843b06313a64-72e2bb75da654e8ab45bba8ac077f28c received from {username='slave'} at 127.0.145.4:52991
I20240503 02:18:35.817101 10016 tablet_copy_service.cc:433] P b85e85e631ab4e04a86bfe5a9f0aa512: ending tablet copy session 8d7c037d50c241b484ee843b06313a64-72e2bb75da654e8ab45bba8ac077f28c on tablet 72e2bb75da654e8ab45bba8ac077f28c with peer 8d7c037d50c241b484ee843b06313a64
I20240503 02:18:35.820823 27300 consensus_queue.cc:579] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Leader has been unable to successfully communicate with peer 03f99420df704238ab300ad6e58d878c for more than 300 seconds (300.377s)
I20240503 02:18:35.824097 27371 ts_tablet_manager.cc:1888] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240503 02:18:35.824290 27371 log.cc:1196] T ecb5c78a22a74728ab40f1da8f90388c P d986fcfb926e47baa7d2d5b99f697334: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/ecb5c78a22a74728ab40f1da8f90388c
I20240503 02:18:35.825508 10261 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:18:35.826190  7814 catalog_manager.cc:4830] TS d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): tablet ecb5c78a22a74728ab40f1da8f90388c (table rebalance_test_table_7 [id=635c8841bff54466a804d93ade171b2f]) successfully deleted
I20240503 02:18:35.831480  8361 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 02:18:35.832429  8232 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 02:18:35.832412 27065 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:18:35.833148 27065 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:18:35.844111 27065 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 12 to 14, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:35.846405  8361 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:35.849038  8408 maintenance_manager.cc:419] P 8d7c037d50c241b484ee843b06313a64: Scheduling UndoDeltaBlockGCOp(72e2bb75da654e8ab45bba8ac077f28c): 4583 bytes on disk
I20240503 02:18:35.849834  8299 maintenance_manager.cc:643] P 8d7c037d50c241b484ee843b06313a64: UndoDeltaBlockGCOp(72e2bb75da654e8ab45bba8ac077f28c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240503 02:18:35.850477  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e966caa651e4e42890be5d79f851a1f with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:35.853217  7828 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 reported cstate change: config changed from index 12 to 14, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 2 leader_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:35.859627 10261 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:35.861634  8361 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240503 02:18:35.863433  8230 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240503 02:18:35.863831 27300 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240503 02:18:35.864665 10194 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e966caa651e4e42890be5d79f851a1f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:35.864833 27300 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:18:35.865796  8230 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, NON_VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } }
I20240503 02:18:35.870429 27300 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:18:35.874409  8361 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:18:35.875382  7816 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7e966caa651e4e42890be5d79f851a1f with cas_config_opid_index 14: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:35.879237  8231 raft_consensus.cc:2932] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } } }
I20240503 02:18:35.880520  7826 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 14 to 15, NON_VOTER d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) added. New cstate: current_term: 2 leader_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:35.888549 27299 raft_consensus.cc:1050] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f: attempting to promote NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c to VOTER
I20240503 02:18:35.889497 27299 consensus_queue.cc:237] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:35.892791  8232 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 LEARNER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:18:35.893456  8100 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:18:35.894589 27398 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:18:35.895020 27401 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:18:35.897634  8361 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:18:35.898204 27299 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:18:35.898094 27400 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.898803  8100 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.900040  8361 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.904229  8232 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.904680  7828 catalog_manager.cc:5484] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: config changed from index 11 to 12, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:35.910984 10126 consensus_queue.cc:237] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:35.912707  8232 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:18:35.912847  8100 raft_consensus.cc:1261] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:18:35.913281 27400 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:18:35.913681 27398 consensus_queue.cc:1035] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:18:35.916131 27401 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.916843  8100 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.916831  8232 raft_consensus.cc:2932] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.921658  7826 catalog_manager.cc:5484] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 12 to 13, VOTER 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) evicted. New cstate: current_term: 2 leader_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.922437  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:35.931083 27300 raft_consensus.cc:1050] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4: attempting to promote NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c to VOTER
I20240503 02:18:35.932029 27300 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 13, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:35.932597  8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 with delete_type TABLET_DATA_TOMBSTONED (TS 8d7c037d50c241b484ee843b06313a64 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:44818
I20240503 02:18:35.933204 27406 tablet_replica.cc:330] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: stopping tablet replica
I20240503 02:18:35.933442 27406 raft_consensus.cc:2229] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:18:35.934019 27406 raft_consensus.cc:2258] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:18:35.934417  9994 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20240503 02:18:35.934562  8232 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 LEARNER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20240503 02:18:35.934859  8100 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20240503 02:18:35.935202 27065 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240503 02:18:35.935294 27401 consensus_queue.cc:579] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Leader has been unable to successfully communicate with peer 03f99420df704238ab300ad6e58d878c for more than 300 seconds (300.043s)
I20240503 02:18:35.935482 27225 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240503 02:18:35.936161 27225 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240503 02:18:35.937537 27406 ts_tablet_manager.cc:1875] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:18:35.943848 10126 consensus_queue.cc:237] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } }
I20240503 02:18:35.946269  8361 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 02:18:35.946945  8100 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 02:18:35.947628 27398 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:18:35.949249 27398 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:18:35.953075 27225 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 LEADER]: Committing config change with OpId 2.17: config changed from index 16 to 17, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.955910  8099 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Committing config change with OpId 2.17: config changed from index 16 to 17, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.956398 27401 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 8 to 11, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:35.958302  8232 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Committing config change with OpId 2.17: config changed from index 16 to 17, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.959213  9995 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Committing config change with OpId 2.17: config changed from index 16 to 17, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.963152  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aeaee03abc644807b8278cfb78e4f02b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:35.962803  7828 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 16 to 17, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:35.962519  7823 catalog_manager.cc:5484] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: config changed from index 8 to 11, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 3 leader_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:35.967705  8361 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 8 to 11, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:35.967811  8101 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 8 to 11, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } }
I20240503 02:18:35.977098 10126 raft_consensus.cc:1203] T b64f66ace86e42f3ac213ba85bcab584 P d826d4dae2aa4e05b2f64c9b17a1126f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12]   Dedup: 2.12->[]
I20240503 02:18:35.981524 10261 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 2.17, Last appended by leader: 13, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:35.983347  9995 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20240503 02:18:35.983471  8232 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Refusing update from remote peer ca05c9a8bee148dab8a2a982b8fa2ea4: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20240503 02:18:35.983911 27300 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240503 02:18:35.984365 27065 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240503 02:18:35.985136 10126 consensus_queue.cc:237] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:18:35.987066 27225 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 2 LEADER]: Committing config change with OpId 2.18: config changed from index 17 to 18, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.987576  8232 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Committing config change with OpId 2.18: config changed from index 17 to 18, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.988134  9994 raft_consensus.cc:2932] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Committing config change with OpId 2.18: config changed from index 17 to 18, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.991369  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f44b2b53de84a87a8757c0d00bfed37 with cas_config_opid_index 17: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 02:18:35.992178 10063 consensus_peers.cc:485] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeaee03abc644807b8278cfb78e4f02b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:35.993171  7826 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 17 to 18, VOTER 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2) evicted. New cstate: current_term: 2 leader_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:35.993803  8101 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 02:18:35.986443 27406 ts_tablet_manager.cc:1888] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240503 02:18:35.993999 27406 log.cc:1196] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 8d7c037d50c241b484ee843b06313a64: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/wals/7e4acc8b5ceb4cd5b0f3708ff4f19298
I20240503 02:18:35.994462 27401 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:18:35.995030  7817 catalog_manager.cc:4830] TS 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298 (table rebalance_test_table_6 [id=938b875abcf9426ead26522c081d6b7c]) successfully deleted
I20240503 02:18:35.995746  8361 raft_consensus.cc:1261] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Refusing update from remote peer d826d4dae2aa4e05b2f64c9b17a1126f: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 02:18:35.996225 27401 consensus_queue.cc:1035] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:18:35.998365 27299 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:18:35.999200  8081 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f44b2b53de84a87a8757c0d00bfed37 with delete_type TABLET_DATA_TOMBSTONED (TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:41102
I20240503 02:18:35.999008  8361 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:18:36.003358  8101 raft_consensus.cc:2932] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:18:36.003687  7814 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet aeaee03abc644807b8278cfb78e4f02b with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:36.004402  7827 catalog_manager.cc:5484] T aeaee03abc644807b8278cfb78e4f02b P d826d4dae2aa4e05b2f64c9b17a1126f reported cstate change: config changed from index 11 to 12, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 3 leader_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:36.015182 27417 tablet_replica.cc:330] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: stopping tablet replica
I20240503 02:18:36.015417 27417 raft_consensus.cc:2229] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:18:36.015650 27417 raft_consensus.cc:2258] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:18:36.016639 27417 ts_tablet_manager.cc:1875] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:18:36.022982 27417 ts_tablet_manager.cc:1888] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.17
I20240503 02:18:36.023111 27417 log.cc:1196] T 0f44b2b53de84a87a8757c0d00bfed37 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/0f44b2b53de84a87a8757c0d00bfed37
I20240503 02:18:36.023811  7816 catalog_manager.cc:4830] TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): tablet 0f44b2b53de84a87a8757c0d00bfed37 (table rebalance_test_table_2 [id=b1c050f6f652453f9abd0ce2412df9e9]) successfully deleted
I20240503 02:18:36.024935  8361 raft_consensus.cc:1203] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.14->[2.15-2.15]   Dedup: 2.15->[]
W20240503 02:18:36.033304  8168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240503 02:18:36.039573 27153 raft_consensus.cc:1050] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: attempting to promote NON_VOTER 2942efc0ecb448d2ba2f6bca20fbb29c to VOTER
I20240503 02:18:36.040249 27153 consensus_queue.cc:237] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:36.042586 10261 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:18:36.042726  8232 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEARNER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 02:18:36.043205 27422 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:18:36.043437  8038 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:36.043475 27153 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:18:36.045953 27422 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:36.046501 10261 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:36.050140  8232 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:36.051043  7830 catalog_manager.cc:5484] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 reported cstate change: config changed from index 10 to 11, 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:36.056576  8101 consensus_queue.cc:237] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:36.058487  8232 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:18:36.058727 10261 raft_consensus.cc:1261] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Refusing update from remote peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 02:18:36.059074 27153 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:18:36.059617 27153 consensus_queue.cc:1035] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:18:36.062211 27153 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:36.062830 10261 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P ca05c9a8bee148dab8a2a982b8fa2ea4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:36.063234  8232 raft_consensus.cc:2932] T 007b59de142c4b95966494ba097b1a09 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } } }
I20240503 02:18:36.066043  7816 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 007b59de142c4b95966494ba097b1a09 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:36.068200  7830 catalog_manager.cc:5484] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 11 to 12, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 1 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:36.178292 27265 raft_consensus.cc:1050] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c: attempting to promote NON_VOTER b85e85e631ab4e04a86bfe5a9f0aa512 to VOTER
I20240503 02:18:36.179257 27265 consensus_queue.cc:237] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:18:36.182245  8101 raft_consensus.cc:1261] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:18:36.182852 27263 consensus_queue.cc:1035] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:18:36.184824  9994 raft_consensus.cc:1261] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 LEARNER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:18:36.185887 27263 consensus_queue.cc:1035] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:18:36.187070  8168 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): Couldn't send request to peer 03f99420df704238ab300ad6e58d878c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:36.188959 27264 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:36.189688  9994 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:36.191931 27372 consensus_queue.cc:579] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Leader has been unable to successfully communicate with peer 03f99420df704238ab300ad6e58d878c for more than 300 seconds (300.502s)
I20240503 02:18:36.193745  8101 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:36.194805  7827 catalog_manager.cc:5484] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 7 to 8, b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03f99420df704238ab300ad6e58d878c" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 44891 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:36.200057  8232 consensus_queue.cc:237] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:18:36.202651  9994 consensus_queue.cc:237] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:18:36.204967  9995 raft_consensus.cc:1261] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:18:36.205487 27263 consensus_queue.cc:1035] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:18:36.205577  8101 raft_consensus.cc:1261] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:18:36.206637 27263 consensus_queue.cc:1035] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:18:36.209679 27263 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:36.212713  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ceae1c7c45a34b24badbbe23cb9e0176 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:36.213392  9994 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:36.213930  7969 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 2 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:18:36.215351 25145 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:18:36.215950  8101 raft_consensus.cc:2932] T ceae1c7c45a34b24badbbe23cb9e0176 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:36.216625  8361 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:18:36.217567 25145 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:18:36.222100  7827 catalog_manager.cc:5484] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 8 to 9, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:36.228509 25145 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:36.233105  7817 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c23b06764ce54acb8388de94aea4cd4b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:36.233446  8361 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:36.233354  7969 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } }
I20240503 02:18:36.234834  7830 catalog_manager.cc:5484] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 9 to 11, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 2 leader_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:36.241786  9994 consensus_queue.cc:237] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:18:36.243824  8361 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:18:36.243898  7969 raft_consensus.cc:1261] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 2 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:18:36.244428 27372 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:18:36.245683 27375 consensus_queue.cc:1035] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:18:36.246651 27372 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:18:36.248719  7969 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P d986fcfb926e47baa7d2d5b99f697334 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
W20240503 02:18:36.250197  9931 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c23b06764ce54acb8388de94aea4cd4b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:36.250655  7817 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c23b06764ce54acb8388de94aea4cd4b with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:36.250458  8361 raft_consensus.cc:2932] T c23b06764ce54acb8388de94aea4cd4b P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } } }
I20240503 02:18:36.253317  7827 catalog_manager.cc:5484] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: config changed from index 11 to 12, NON_VOTER ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) added. New cstate: current_term: 2 leader_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:18:36.291030  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ceae1c7c45a34b24badbbe23cb9e0176: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:18:36.302960 27479 ts_tablet_manager.cc:902] T 7e966caa651e4e42890be5d79f851a1f P d826d4dae2aa4e05b2f64c9b17a1126f: Initiating tablet copy from peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
F20240503 02:18:36.303321 27479 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-32e527fa9b2f441cb5668435681b4ff6.queue_time_us, tablet-download-pool-32e527fa9b2f441cb5668435681b4ff6.queue_time_us) (tablet-download-pool-32e527fa9b2f441cb5668435681b4ff6.run_cpu_time_us, tablet-download-pool-32e527fa9b2f441cb5668435681b4ff6.run_cpu_time_us) (tablet-download-pool-32e527fa9b2f441cb5668435681b4ff6.run_wall_time_us, tablet-download-pool-32e527fa9b2f441cb5668435681b4ff6.run_wall_time_us) (tablet-download-pool-402de43677c44deab8793da1895d4fb2.queue_time_us, tablet-download-pool-402de43677c44deab8793da1895d4fb2.queue_time_us) (tablet-download-pool-402de43677c44deab8793da1895d4fb2.run_cpu_time_us, tablet-download-pool-402de43677c44deab8793da1895d4fb2.run_cpu_time_us) (tablet-download-pool-402de43677c44deab8793da1895d4fb2.run_wall_time_us, tablet-download-pool-402de43677c44deab8793da1895d4fb2.run_wall_time_us) (tablet-download-pool-4f2cedba9f9a4e709623fceeffcfd095.queue_time_us, tablet-download-pool-4f2cedba9f9a4e709623fceeffcfd095.queue_time_us) (tablet-download-pool-4f2cedba9f9a4e709623fceeffcfd095.run_cpu_time_us, tablet-download-pool-4f2cedba9f9a4e709623fceeffcfd095.run_cpu_time_us) (tablet-download-pool-4f2cedba9f9a4e709623fceeffcfd095.run_wall_time_us, tablet-download-pool-4f2cedba9f9a4e709623fceeffcfd095.run_wall_time_us) (tablet-download-pool-5ba1f505010b403c91d3870cee19952a.queue_time_us, tablet-download-pool-5ba1f505010b403c91d3870cee19952a.queue_time_us) (tablet-download-pool-5ba1f505010b403c91d3870cee19952a.run_cpu_time_us, tablet-download-pool-5ba1f505010b403c91d3870cee19952a.run_cpu_time_us) (tablet-download-pool-5ba1f505010b403c91d3870cee19952a.run_wall_time_us, tablet-download-pool-5ba1f505010b403c91d3870cee19952a.run_wall_time_us) (tablet-download-pool-6043da7d2e584af885ff6a5bf8f06912.queue_time_us, tablet-download-pool-6043da7d2e584af885ff6a5bf8f06912.queue_time_us) (tablet-download-pool-6043da7d2e584af885ff6a5bf8f06912.run_cpu_time_us, tablet-download-pool-6043da7d2e584af885ff6a5bf8f06912.run_cpu_time_us) (tablet-download-pool-6043da7d2e584af885ff6a5bf8f06912.run_wall_time_us, tablet-download-pool-6043da7d2e584af885ff6a5bf8f06912.run_wall_time_us) (tablet-download-pool-72e2bb75da654e8ab45bba8ac077f28c.queue_time_us, tablet-download-pool-72e2bb75da654e8ab45bba8ac077f28c.queue_time_us) (tablet-download-pool-72e2bb75da654e8ab45bba8ac077f28c.run_cpu_time_us, tablet-download-pool-72e2bb75da654e8ab45bba8ac077f28c.run_cpu_time_us) (tablet-download-pool-72e2bb75da654e8ab45bba8ac077f28c.run_wall_time_us, tablet-download-pool-72e2bb75da654e8ab45bba8ac077f28c.run_wall_time_us) (tablet-download-pool-7e4acc8b5ceb4cd5b0f3708ff4f19298.queue_time_us, tablet-download-pool-7e4acc8b5ceb4cd5b0f3708ff4f19298.queue_time_us) (tablet-download-pool-7e4acc8b5ceb4cd5b0f3708ff4f19298.run_cpu_time_us, tablet-download-pool-7e4acc8b5ceb4cd5b0f3708ff4f19298.run_cpu_time_us) (tablet-download-pool-7e4acc8b5ceb4cd5b0f3708ff4f19298.run_wall_time_us, tablet-download-pool-7e4acc8b5ceb4cd5b0f3708ff4f19298.run_wall_time_us) (tablet-download-pool-7e966caa651e4e42890be5d79f851a1f.queue_time_us, tablet-download-pool-7e966caa651e4e42890be5d79f851a1f.queue_time_us) (tablet-download-pool-8a8affdde4d8417f95f0585dd01e7d59.queue_time_us, tablet-download-pool-8a8affdde4d8417f95f0585dd01e7d59.queue_time_us) (tablet-download-pool-8a8affdde4d8417f95f0585dd01e7d59.run_cpu_time_us, tablet-download-pool-8a8affdde4d8417f95f0585dd01e7d59.run_cpu_time_us) (tablet-download-pool-8a8affdde4d8417f95f0585dd01e7d59.run_wall_time_us, tablet-download-pool-8a8affdde4d8417f95f0585dd01e7d59.run_wall_time_us) (tablet-download-pool-aeaee03abc644807b8278cfb78e4f02b.queue_time_us, tablet-download-pool-aeaee03abc644807b8278cfb78e4f02b.queue_time_us) (tablet-download-pool-aeaee03abc644807b8278cfb78e4f02b.run_cpu_time_us, tablet-download-pool-aeaee03abc644807b8278cfb78e4f02b.run_cpu_time_us) (tablet-download-pool-aeaee03abc644807b8278cfb78e4f02b.run_wall_time_us, tablet-download-pool-aeaee03abc644807b8278cfb78e4f02b.run_wall_time_us) (tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.queue_time_us, tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.queue_time_us) (tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.run_cpu_time_us, tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.run_cpu_time_us) (tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.run_wall_time_us, tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.run_wall_time_us) (tablet-download-pool-d3d4b912095244de816d3089a31b8649.queue_time_us, tablet-download-pool-d3d4b912095244de816d3089a31b8649.queue_time_us) (tablet-download-pool-d3d4b912095244de816d3089a31b8649.run_cpu_time_us, tablet-download-pool-d3d4b912095244de816d3089a31b8649.run_cpu_time_us) (tablet-download-pool-d3d4b912095244de816d3089a31b8649.run_wall_time_us, tablet-download-pool-d3d4b912095244de816d3089a31b8649.run_wall_time_us) (tablet-download-pool-de9c2a3396864696af88be1ca12ee0f1.queue_time_us, tablet-download-pool-de9c2a3396864696af88be1ca12ee0f1.queue_time_us) (tablet-download-pool-de9c2a3396864696af88be1ca12ee0f1.run_cpu_time_us, tablet-download-pool-de9c2a3396864696af88be1ca12ee0f1.run_cpu_time_us) (tablet-download-pool-de9c2a3396864696af88be1ca12ee0f1.run_wall_time_us, tablet-download-pool-de9c2a3396864696af88be1ca12ee0f1.run_wall_time_us) (tablet-download-pool-ecb5c78a22a74728ab40f1da8f90388c.queue_time_us, tablet-download-pool-ecb5c78a22a74728ab40f1da8f90388c.queue_time_us) (tablet-download-pool-ecb5c78a22a74728ab40f1da8f90388c.run_cpu_time_us, tablet-download-pool-ecb5c78a22a74728ab40f1da8f90388c.run_cpu_time_us) (tablet-download-pool-ecb5c78a22a74728ab40f1da8f90388c.run_wall_time_us, tablet-download-pool-ecb5c78a22a74728ab40f1da8f90388c.run_wall_time_us) (tablet-download-pool-f14607425da84bb4b162a5789daa844c.queue_time_us, tablet-download-pool-f14607425da84bb4b162a5789daa844c.queue_time_us) (tablet-download-pool-f14607425da84bb4b162a5789daa844c.run_cpu_time_us, tablet-download-pool-f14607425da84bb4b162a5789daa844c.run_cpu_time_us) (tablet-download-pool-f14607425da84bb4b162a5789daa844c.run_wall_time_us, tablet-download-pool-f14607425da84bb4b162a5789daa844c.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7f610d375dcd  google::LogMessage::Fail() at ??:0
    @     0x7f610d379b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f610d3757cc  google::LogMessage::Flush() at ??:0
    @     0x7f610d376f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f61100b6742  kudu::TraceMetrics::InternName() at ??:0
    @     0x7f6110082f8b  kudu::ThreadPool::ThreadPool() at ??:0
    @     0x7f611007ccd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7f611f4d08de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7f611f4e8daf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
    @     0x7f611f613480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
    @     0x7f611f64ba8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 02:18:36.372447 27490 ts_tablet_manager.cc:902] T aeaee03abc644807b8278cfb78e4f02b P ca05c9a8bee148dab8a2a982b8fa2ea4: Initiating tablet copy from peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
F20240503 02:18:36.372793 27490 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-007b59de142c4b95966494ba097b1a09.queue_time_us, tablet-download-pool-007b59de142c4b95966494ba097b1a09.queue_time_us) (tablet-download-pool-007b59de142c4b95966494ba097b1a09.run_cpu_time_us, tablet-download-pool-007b59de142c4b95966494ba097b1a09.run_cpu_time_us) (tablet-download-pool-007b59de142c4b95966494ba097b1a09.run_wall_time_us, tablet-download-pool-007b59de142c4b95966494ba097b1a09.run_wall_time_us) (tablet-download-pool-0f44b2b53de84a87a8757c0d00bfed37.queue_time_us, tablet-download-pool-0f44b2b53de84a87a8757c0d00bfed37.queue_time_us) (tablet-download-pool-0f44b2b53de84a87a8757c0d00bfed37.run_cpu_time_us, tablet-download-pool-0f44b2b53de84a87a8757c0d00bfed37.run_cpu_time_us) (tablet-download-pool-0f44b2b53de84a87a8757c0d00bfed37.run_wall_time_us, tablet-download-pool-0f44b2b53de84a87a8757c0d00bfed37.run_wall_time_us) (tablet-download-pool-2d69c71d9dfd47a1a2429e7caf5e33ac.queue_time_us, tablet-download-pool-2d69c71d9dfd47a1a2429e7caf5e33ac.queue_time_us) (tablet-download-pool-2d69c71d9dfd47a1a2429e7caf5e33ac.run_cpu_time_us, tablet-download-pool-2d69c71d9dfd47a1a2429e7caf5e33ac.run_cpu_time_us) (tablet-download-pool-2d69c71d9dfd47a1a2429e7caf5e33ac.run_wall_time_us, tablet-download-pool-2d69c71d9dfd47a1a2429e7caf5e33ac.run_wall_time_us) (tablet-download-pool-3dbfc7f12fc142f98b1d1c026fd49f5a.queue_time_us, tablet-download-pool-3dbfc7f12fc142f98b1d1c026fd49f5a.queue_time_us) (tablet-download-pool-3dbfc7f12fc142f98b1d1c026fd49f5a.run_cpu_time_us, tablet-download-pool-3dbfc7f12fc142f98b1d1c026fd49f5a.run_cpu_time_us) (tablet-download-pool-3dbfc7f12fc142f98b1d1c026fd49f5a.run_wall_time_us, tablet-download-pool-3dbfc7f12fc142f98b1d1c026fd49f5a.run_wall_time_us) (tablet-download-pool-4f2cedba9f9a4e709623fceeffcfd095.queue_time_us, tablet-download-pool-4f2cedba9f9a4e709623fceeffcfd095.queue_time_us) (tablet-download-pool-4f2cedba9f9a4e709623fceeffcfd095.run_cpu_time_us, tablet-download-pool-4f2cedba9f9a4e709623fceeffcfd095.run_cpu_time_us) (tablet-download-pool-4f2cedba9f9a4e709623fceeffcfd095.run_wall_time_us, tablet-download-pool-4f2cedba9f9a4e709623fceeffcfd095.run_wall_time_us) (tablet-download-pool-5ea56405515a40f785726bd58ec3d26a.queue_time_us, tablet-download-pool-5ea56405515a40f785726bd58ec3d26a.queue_time_us) (tablet-download-pool-5ea56405515a40f785726bd58ec3d26a.run_cpu_time_us, tablet-download-pool-5ea56405515a40f785726bd58ec3d26a.run_cpu_time_us) (tablet-download-pool-5ea56405515a40f785726bd58ec3d26a.run_wall_time_us, tablet-download-pool-5ea56405515a40f785726bd58ec3d26a.run_wall_time_us) (tablet-download-pool-7e966caa651e4e42890be5d79f851a1f.queue_time_us, tablet-download-pool-7e966caa651e4e42890be5d79f851a1f.queue_time_us) (tablet-download-pool-7e966caa651e4e42890be5d79f851a1f.run_cpu_time_us, tablet-download-pool-7e966caa651e4e42890be5d79f851a1f.run_cpu_time_us) (tablet-download-pool-7e966caa651e4e42890be5d79f851a1f.run_wall_time_us, tablet-download-pool-7e966caa651e4e42890be5d79f851a1f.run_wall_time_us) (tablet-download-pool-834aaba965c8443c862e98a070e36cac.queue_time_us, tablet-download-pool-834aaba965c8443c862e98a070e36cac.queue_time_us) (tablet-download-pool-834aaba965c8443c862e98a070e36cac.run_cpu_time_us, tablet-download-pool-834aaba965c8443c862e98a070e36cac.run_cpu_time_us) (tablet-download-pool-834aaba965c8443c862e98a070e36cac.run_wall_time_us, tablet-download-pool-834aaba965c8443c862e98a070e36cac.run_wall_time_us) (tablet-download-pool-aeaee03abc644807b8278cfb78e4f02b.queue_time_us, tablet-download-pool-aeaee03abc644807b8278cfb78e4f02b.queue_time_us) (tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.queue_time_us, tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.queue_time_us) (tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.run_cpu_time_us, tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.run_cpu_time_us) (tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.run_wall_time_us, tablet-download-pool-b64f66ace86e42f3ac213ba85bcab584.run_wall_time_us) (tablet-download-pool-c45c5a14b85447c78fe2a27be4392c88.queue_time_us, tablet-download-pool-c45c5a14b85447c78fe2a27be4392c88.queue_time_us) (tablet-download-pool-c45c5a14b85447c78fe2a27be4392c88.run_cpu_time_us, tablet-download-pool-c45c5a14b85447c78fe2a27be4392c88.run_cpu_time_us) (tablet-download-pool-c45c5a14b85447c78fe2a27be4392c88.run_wall_time_us, tablet-download-pool-c45c5a14b85447c78fe2a27be4392c88.run_wall_time_us) (tablet-download-pool-cab70b62bdc747a9b653bffde680f37e.queue_time_us, tablet-download-pool-cab70b62bdc747a9b653bffde680f37e.queue_time_us) (tablet-download-pool-cab70b62bdc747a9b653bffde680f37e.run_cpu_time_us, tablet-download-pool-cab70b62bdc747a9b653bffde680f37e.run_cpu_time_us) (tablet-download-pool-cab70b62bdc747a9b653bffde680f37e.run_wall_time_us, tablet-download-pool-cab70b62bdc747a9b653bffde680f37e.run_wall_time_us) (tablet-download-pool-de9c2a3396864696af88be1ca12ee0f1.queue_time_us, tablet-download-pool-de9c2a3396864696af88be1ca12ee0f1.queue_time_us) (tablet-download-pool-de9c2a3396864696af88be1ca12ee0f1.run_cpu_time_us, tablet-download-pool-de9c2a3396864696af88be1ca12ee0f1.run_cpu_time_us) (tablet-download-pool-de9c2a3396864696af88be1ca12ee0f1.run_wall_time_us, tablet-download-pool-de9c2a3396864696af88be1ca12ee0f1.run_wall_time_us) (tablet-download-pool-ecb5c78a22a74728ab40f1da8f90388c.queue_time_us, tablet-download-pool-ecb5c78a22a74728ab40f1da8f90388c.queue_time_us) (tablet-download-pool-ecb5c78a22a74728ab40f1da8f90388c.run_cpu_time_us, tablet-download-pool-ecb5c78a22a74728ab40f1da8f90388c.run_cpu_time_us) (tablet-download-pool-ecb5c78a22a74728ab40f1da8f90388c.run_wall_time_us, tablet-download-pool-ecb5c78a22a74728ab40f1da8f90388c.run_wall_time_us) (tablet-download-pool-f14607425da84bb4b162a5789daa844c.queue_time_us, tablet-download-pool-f14607425da84bb4b162a5789daa844c.queue_time_us) (tablet-download-pool-f14607425da84bb4b162a5789daa844c.run_cpu_time_us, tablet-download-pool-f14607425da84bb4b162a5789daa844c.run_cpu_time_us) (tablet-download-pool-f14607425da84bb4b162a5789daa844c.run_wall_time_us, tablet-download-pool-f14607425da84bb4b162a5789daa844c.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7f34d2bdcdcd  google::LogMessage::Fail() at ??:0
    @     0x7f34d2be0b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f34d2bdc7cc  google::LogMessage::Flush() at ??:0
    @     0x7f34d2bddf59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f611f62e981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
    @     0x7f34d591d742  kudu::TraceMetrics::InternName() at ??:0
    @     0x7f34d58e9f8b  kudu::ThreadPool::ThreadPool() at ??:0
    @     0x7f611f62e70e  std::_Function_handler<>::_M_invoke() at ??:0
    @     0x7f34d58e3cd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f6110086b75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7f34e4d378de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7f611008a19c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7f611008a019  std::_Function_handler<>::_M_invoke() at ??:0
    @     0x7f34e4d4fdaf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f34e4e7a480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
    @     0x7f611006501e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f34e4eb2a8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
    @     0x7f61133ea6db  start_thread at ??:0
    @     0x7f610c22871f  clone at ??:0
    @     0x7f34e4e95981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
    @     0x7f34e4e9570e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f34d58edb75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7f34d58f119c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7f34d58f1019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f34d58cc01e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f34d8c516db  start_thread at ??:0
    @     0x7f34d1a8f71f  clone at ??:0
I20240503 02:18:36.489620 25145 raft_consensus.cc:1050] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512: attempting to promote NON_VOTER 8d7c037d50c241b484ee843b06313a64 to VOTER
I20240503 02:18:36.490733 25145 consensus_queue.cc:237] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 12, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } }
I20240503 02:18:36.493288  8361 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 2 LEARNER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20240503 02:18:36.493858  8101 raft_consensus.cc:1261] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20240503 02:18:36.494453 27375 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 02:18:36.494812 27372 consensus_queue.cc:1035] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 02:18:36.498767 27372 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } } }
I20240503 02:18:36.503019  8101 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } } }
I20240503 02:18:36.508632  8361 raft_consensus.cc:2932] T 72e2bb75da654e8ab45bba8ac077f28c P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } } }
I20240503 02:18:36.509333  7826 catalog_manager.cc:5484] T 72e2bb75da654e8ab45bba8ac077f28c P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: config changed from index 15 to 16, 8d7c037d50c241b484ee843b06313a64 (127.0.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { promote: false } } }
W20240503 02:18:36.782138  9931 connection.cc:535] client connection to 127.0.145.8:38885 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:36.782574  8297 connection.cc:535] client connection to 127.0.145.8:38885 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:36.782616  8037 connection.cc:535] client connection to 127.0.145.8:38885 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:36.782651  8167 connection.cc:535] client connection to 127.0.145.8:38885 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:36.782721  8297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 8 similar messages]
W20240503 02:18:36.790077  7906 connection.cc:535] client connection to 127.0.145.8:38885 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:36.790401  7906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:36.796478 13512 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:18:36.783399 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.145.8:38885 (local address 127.0.145.3:57723)
0503 02:18:36.783641 (+   242us) negotiation.cc:107] Waiting for socket to connect
0503 02:18:36.783668 (+    27us) client_negotiation.cc:174] Beginning negotiation
0503 02:18:36.783843 (+   175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:18:36.795918 (+ 12075us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":54}
W20240503 02:18:36.796684  8167 consensus_peers.cc:485] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.796836  8167 consensus_peers.cc:485] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.807185  8037 consensus_peers.cc:485] T 007b59de142c4b95966494ba097b1a09 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.807475  8037 consensus_peers.cc:485] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.807874  7906 consensus_peers.cc:485] T c45c5a14b85447c78fe2a27be4392c88 P d986fcfb926e47baa7d2d5b99f697334 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.808085  7906 consensus_peers.cc:485] T 5ea56405515a40f785726bd58ec3d26a P d986fcfb926e47baa7d2d5b99f697334 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.808107  9931 consensus_peers.cc:462] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
W20240503 02:18:36.809384 27558 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:18:36.794247 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.145.8:38885 (local address 127.0.145.4:55145)
0503 02:18:36.808665 (+ 14418us) negotiation.cc:107] Waiting for socket to connect
0503 02:18:36.808817 (+   152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":14277,"thread_start_us":14304,"threads_started":1}
W20240503 02:18:36.809717  8297 consensus_peers.cc:485] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.810227  8297 consensus_peers.cc:485] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.811189  8297 consensus_peers.cc:485] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.816366  9929 connection.cc:535] client connection to 127.0.145.7:39415 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:36.816483  8295 connection.cc:535] client connection to 127.0.145.7:39415 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:36.816494  8165 connection.cc:535] client connection to 127.0.145.7:39415 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:36.816926  8035 connection.cc:535] client connection to 127.0.145.7:39415 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:36.816990  7904 connection.cc:535] client connection to 127.0.145.7:39415 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:36.818812 27559 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:18:36.817058 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.145.7:39415 (local address 127.0.145.6:44463)
0503 02:18:36.817377 (+   319us) negotiation.cc:107] Waiting for socket to connect
0503 02:18:36.817397 (+    20us) client_negotiation.cc:174] Beginning negotiation
0503 02:18:36.817513 (+   116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:18:36.818530 (+  1017us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":204}
W20240503 02:18:36.819061 27561 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:18:36.817764 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.145.7:39415 (local address 127.0.145.2:37371)
0503 02:18:36.818020 (+   256us) negotiation.cc:107] Waiting for socket to connect
0503 02:18:36.818041 (+    21us) client_negotiation.cc:174] Beginning negotiation
0503 02:18:36.818207 (+   166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:18:36.818510 (+   303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":67}
W20240503 02:18:36.819670 13512 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:18:36.817872 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.145.7:39415 (local address 127.0.145.3:34021)
0503 02:18:36.818030 (+   158us) negotiation.cc:107] Waiting for socket to connect
0503 02:18:36.818046 (+    16us) client_negotiation.cc:174] Beginning negotiation
0503 02:18:36.818151 (+   105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:18:36.819448 (+  1297us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":43}
W20240503 02:18:36.819931 27560 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:18:36.818028 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.145.7:39415 (local address 127.0.145.1:36965)
0503 02:18:36.818268 (+   240us) negotiation.cc:107] Waiting for socket to connect
0503 02:18:36.818284 (+    16us) client_negotiation.cc:174] Beginning negotiation
0503 02:18:36.818400 (+   116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:18:36.819422 (+  1022us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":137}
W20240503 02:18:36.820112  8165 consensus_peers.cc:485] T f14607425da84bb4b162a5789daa844c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.820165  7904 consensus_peers.cc:485] T 4f2cedba9f9a4e709623fceeffcfd095 P d986fcfb926e47baa7d2d5b99f697334 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.820219  8165 consensus_peers.cc:485] T ecb5c78a22a74728ab40f1da8f90388c P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.820302  8165 consensus_peers.cc:485] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.820384  8165 consensus_peers.cc:485] T 32e527fa9b2f441cb5668435681b4ff6 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.820176  9929 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.820497 27558 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:18:36.818657 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.145.7:39415 (local address 127.0.145.4:42501)
0503 02:18:36.820269 (+  1612us) negotiation.cc:107] Waiting for socket to connect
0503 02:18:36.820353 (+    84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":1507}
W20240503 02:18:36.820988  8295 consensus_peers.cc:485] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.821121  8295 consensus_peers.cc:485] T d3d4b912095244de816d3089a31b8649 P 8d7c037d50c241b484ee843b06313a64 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.820742  8035 consensus_peers.cc:485] T 8a8affdde4d8417f95f0585dd01e7d59 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.864379  7906 consensus_peers.cc:485] T cab70b62bdc747a9b653bffde680f37e P d986fcfb926e47baa7d2d5b99f697334 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:36.897589  8295 consensus_peers.cc:485] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:37.002503  7904 consensus_peers.cc:485] T 5ba1f505010b403c91d3870cee19952a P d986fcfb926e47baa7d2d5b99f697334 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:37.035091  8211 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:37.037901  9976 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:18:37.059149  7950 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:37.073159  8341 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:37.073873  8211 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:37.075860  8341 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:37.100628  9976 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:18:37.107707  8081 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:18:37.109272  8081 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:18:37.118753 27581 consensus_queue.cc:579] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Leader has been unable to successfully communicate with peer 03f99420df704238ab300ad6e58d878c for more than 300 seconds (300.537s)
I20240503 02:18:37.125715  8232 consensus_queue.cc:237] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } }
I20240503 02:18:37.148571  8101 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:18:37.149470 27581 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:18:37.157166  8361 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:18:37.158836 27579 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:18:37.161619  9931 consensus_peers.cc:462] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:37.169930 27579 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:18:37.174521  8361 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:18:37.176442  7814 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2448e12efbd44a90b1f8bce9519e73e0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:37.178936  7828 catalog_manager.cc:5484] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 8 to 9, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:37.197450  8099 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 03f99420df704238ab300ad6e58d878c (127.0.145.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } }
I20240503 02:18:37.241438  7950 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:37.263161  8232 consensus_queue.cc:237] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } }
I20240503 02:18:37.268467  8101 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:18:37.269062  8361 raft_consensus.cc:1261] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:18:37.269696 27264 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:18:37.270048 27581 consensus_queue.cc:1035] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 02:18:37.272022  8165 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Couldn't send request to peer d986fcfb926e47baa7d2d5b99f697334. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2448e12efbd44a90b1f8bce9519e73e0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:37.273720 27579 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } } }
I20240503 02:18:37.275085  8101 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } } }
I20240503 02:18:37.278249  7814 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2448e12efbd44a90b1f8bce9519e73e0 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:37.277915  8361 raft_consensus.cc:2932] T 2448e12efbd44a90b1f8bce9519e73e0 P 8d7c037d50c241b484ee843b06313a64 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } } }
I20240503 02:18:37.280874  7828 catalog_manager.cc:5484] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: config changed from index 9 to 10, NON_VOTER d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1) added. New cstate: current_term: 1 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:18:37.321763  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2448e12efbd44a90b1f8bce9519e73e0: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:18:37.404709 27643 raft_consensus.cc:482] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca05c9a8bee148dab8a2a982b8fa2ea4)
I20240503 02:18:37.404939 27643 raft_consensus.cc:504] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:37.405910 27643 leader_election.cc:288] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
W20240503 02:18:37.408228  8037 leader_election.cc:334] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:37.413206  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64f66ace86e42f3ac213ba85bcab584" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:18:37.413892  8038 leader_election.cc:302] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 8d7c037d50c241b484ee843b06313a64, ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:18:37.414214 27643 raft_consensus.cc:2726] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:18:37.417948 27579 raft_consensus.cc:482] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d826d4dae2aa4e05b2f64c9b17a1126f)
I20240503 02:18:37.418192 27579 raft_consensus.cc:504] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:37.418965 27579 leader_election.cc:288] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
I20240503 02:18:37.419461  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
W20240503 02:18:37.421025  8165 leader_election.cc:334] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111)
I20240503 02:18:37.421173  8165 leader_election.cc:302] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c; no voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:18:37.421451 27579 raft_consensus.cc:2726] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:18:37.488277 27579 raft_consensus.cc:482] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca05c9a8bee148dab8a2a982b8fa2ea4)
I20240503 02:18:37.488492 27579 raft_consensus.cc:504] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:37.489321 27372 raft_consensus.cc:482] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca05c9a8bee148dab8a2a982b8fa2ea4)
I20240503 02:18:37.489508 27372 raft_consensus.cc:504] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:37.490286 27372 leader_election.cc:288] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
W20240503 02:18:37.490702  8167 leader_election.cc:334] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:37.491531  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f44b2b53de84a87a8757c0d00bfed37" candidate_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" candidate_term: 3 candidate_status { last_received { term: 2 index: 18 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:18:37.491730  8232 raft_consensus.cc:2454] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b85e85e631ab4e04a86bfe5a9f0aa512 in term 2.
W20240503 02:18:37.492202  9931 leader_election.cc:334] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:37.492388  9929 leader_election.cc:302] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, b85e85e631ab4e04a86bfe5a9f0aa512; no voters: ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:18:37.492657 27372 raft_consensus.cc:2781] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:18:37.492754 27372 raft_consensus.cc:482] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ca05c9a8bee148dab8a2a982b8fa2ea4)
I20240503 02:18:37.492827 27372 raft_consensus.cc:3037] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:18:37.492820 27579 leader_election.cc:288] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885), b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6:40397)
I20240503 02:18:37.493345  9994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f44b2b53de84a87a8757c0d00bfed37" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 3 candidate_status { last_received { term: 2 index: 18 } } ignore_live_leader: false dest_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" is_pre_election: true
I20240503 02:18:37.495303 27372 raft_consensus.cc:504] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:37.495750  9994 raft_consensus.cc:2379] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2942efc0ecb448d2ba2f6bca20fbb29c in current term 3: Already voted for candidate b85e85e631ab4e04a86bfe5a9f0aa512 in this term.
I20240503 02:18:37.496536  8168 leader_election.cc:302] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c; no voters: b85e85e631ab4e04a86bfe5a9f0aa512, ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:18:37.496695 27372 leader_election.cc:288] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 3 election: Requested vote from peers ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:18:37.497505  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f44b2b53de84a87a8757c0d00bfed37" candidate_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" candidate_term: 3 candidate_status { last_received { term: 2 index: 18 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
W20240503 02:18:37.497609  9931 leader_election.cc:334] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:37.497684  8232 raft_consensus.cc:3037] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:18:37.500139 27613 raft_consensus.cc:482] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d826d4dae2aa4e05b2f64c9b17a1126f)
I20240503 02:18:37.500319 27613 raft_consensus.cc:504] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:18:37.500792  8232 raft_consensus.cc:2454] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b85e85e631ab4e04a86bfe5a9f0aa512 in term 3.
I20240503 02:18:37.501145 27613 leader_election.cc:288] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
I20240503 02:18:37.501565  9929 leader_election.cc:302] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, b85e85e631ab4e04a86bfe5a9f0aa512; no voters: ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:18:37.501910 27372 raft_consensus.cc:2781] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:18:37.504004 27643 raft_consensus.cc:482] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d826d4dae2aa4e05b2f64c9b17a1126f)
W20240503 02:18:37.504109  8295 leader_election.cc:334] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111)
I20240503 02:18:37.504160 27643 raft_consensus.cc:504] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:18:37.504758  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:18:37.504992  8101 raft_consensus.cc:2454] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 3.
I20240503 02:18:37.504776 27643 leader_election.cc:288] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
I20240503 02:18:37.505734  8297 leader_election.cc:302] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:18:37.506242  8035 leader_election.cc:334] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111)
I20240503 02:18:37.506367  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:18:37.506485 27613 raft_consensus.cc:2781] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240503 02:18:37.506630  8361 raft_consensus.cc:2454] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 3.
I20240503 02:18:37.506757 27613 raft_consensus.cc:482] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d826d4dae2aa4e05b2f64c9b17a1126f)
I20240503 02:18:37.506850 27613 raft_consensus.cc:3037] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:18:37.507164  8038 leader_election.cc:302] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:18:37.507676 27643 raft_consensus.cc:2781] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240503 02:18:37.507767 27643 raft_consensus.cc:482] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Starting leader election (detected failure of leader d826d4dae2aa4e05b2f64c9b17a1126f)
I20240503 02:18:37.507838 27643 raft_consensus.cc:3037] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:18:37.509624 27613 raft_consensus.cc:504] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:18:37.510668  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:18:37.510784 27579 raft_consensus.cc:2726] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:18:37.511088 27613 leader_election.cc:288] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 4 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
W20240503 02:18:37.512547  8295 leader_election.cc:334] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111)
I20240503 02:18:37.512774 27643 raft_consensus.cc:504] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:18:37.513183  8101 raft_consensus.cc:2379] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8d7c037d50c241b484ee843b06313a64 in current term 4: Already voted for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in this term.
I20240503 02:18:37.513885  8297 leader_election.cc:302] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64; no voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:18:37.514185 27613 raft_consensus.cc:2726] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:18:37.514997  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeaee03abc644807b8278cfb78e4f02b" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:18:37.515239  8361 raft_consensus.cc:2379] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in current term 4: Already voted for candidate 8d7c037d50c241b484ee843b06313a64 in this term.
I20240503 02:18:37.517405 27643 leader_election.cc:288] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 4 election: Requested vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
I20240503 02:18:37.518254 27372 raft_consensus.cc:686] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 3 LEADER]: Becoming Leader. State: Replica: b85e85e631ab4e04a86bfe5a9f0aa512, State: Running, Role: LEADER
I20240503 02:18:37.518589 27372 consensus_queue.cc:237] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
W20240503 02:18:37.518862  8035 leader_election.cc:334] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111)
I20240503 02:18:37.518975  8035 leader_election.cc:302] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 8d7c037d50c241b484ee843b06313a64, d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:18:37.519454 27643 raft_consensus.cc:2726] T aeaee03abc644807b8278cfb78e4f02b P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 02:18:37.521888  7830 catalog_manager.cc:5484] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 reported cstate change: term changed from 2 to 3, leader changed from ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) to b85e85e631ab4e04a86bfe5a9f0aa512 (127.0.145.6). New cstate: current_term: 3 leader_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:18:37.555461  9931 consensus_peers.cc:462] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:37.570161 27613 raft_consensus.cc:482] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca05c9a8bee148dab8a2a982b8fa2ea4)
I20240503 02:18:37.570415 27613 raft_consensus.cc:504] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:37.571394 27613 leader_election.cc:288] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
I20240503 02:18:37.572815  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64f66ace86e42f3ac213ba85bcab584" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" is_pre_election: true
I20240503 02:18:37.573093  8101 raft_consensus.cc:2454] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 2.
I20240503 02:18:37.574805  8297 leader_election.cc:302] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: 
I20240503 02:18:37.575525 27613 raft_consensus.cc:2781] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:18:37.575646 27613 raft_consensus.cc:482] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ca05c9a8bee148dab8a2a982b8fa2ea4)
I20240503 02:18:37.575733 27613 raft_consensus.cc:3037] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:18:37.578923  8297 leader_election.cc:334] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:37.580515 27613 raft_consensus.cc:504] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:37.584113  8101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64f66ace86e42f3ac213ba85bcab584" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d"
I20240503 02:18:37.584342  8101 raft_consensus.cc:3037] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:18:37.585110 27613 leader_election.cc:288] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 3 election: Requested vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
W20240503 02:18:37.586742  8297 leader_election.cc:334] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:37.586889  8101 raft_consensus.cc:2454] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d7c037d50c241b484ee843b06313a64 in term 3.
I20240503 02:18:37.587333  8297 leader_election.cc:302] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:18:37.587632 27613 raft_consensus.cc:2781] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:18:37.588150 27613 raft_consensus.cc:686] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [term 3 LEADER]: Becoming Leader. State: Replica: 8d7c037d50c241b484ee843b06313a64, State: Running, Role: LEADER
I20240503 02:18:37.588550 27613 consensus_queue.cc:237] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:37.592555  7830 catalog_manager.cc:5484] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 reported cstate change: term changed from 2 to 3, leader changed from ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) to 8d7c037d50c241b484ee843b06313a64 (127.0.145.4). New cstate: current_term: 3 leader_uuid: "8d7c037d50c241b484ee843b06313a64" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:37.697500 27659 ts_tablet_manager.cc:902] T 2448e12efbd44a90b1f8bce9519e73e0 P d986fcfb926e47baa7d2d5b99f697334: Initiating tablet copy from peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:18:37.708698 10313 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240503 02:13:14.245774 10315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:14.254341 10315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:14.304510 10315 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240503 02:13:15.787160 10474 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.0.145.4:35377, user_credentials={real_user=slave}} blocked reactor thread for 56408.5us
I20240503 02:13:17.830346 10315 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:13:21.095615 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b1c050f6f652453f9abd0ce2412df9e9: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:21.102540 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:21.102794 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7d72eaa895849fcb02235c1b6e8d08f: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:21.102926 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:21.103041 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:21.107314 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:21.107565 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:21.107692 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:21.107810 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:21.107916 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7d72eaa895849fcb02235c1b6e8d08f: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:21.108004 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:21.108151 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8da59f26450d4d25a86bc61be2943c37: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:21.108273 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:21.108389 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:21.108570 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b1c050f6f652453f9abd0ce2412df9e9: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:21.117354 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:21.117533 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:21.117662 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:21.117770 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7d72eaa895849fcb02235c1b6e8d08f: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:21.117965 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:21.118085 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:21.671538 10315 rebalancer_tool.cc:1074] tablet 5ba1f505010b403c91d3870cee19952a: '8d7c037d50c241b484ee843b06313a64' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move scheduled
I20240503 02:13:21.909193 10315 rebalancer_tool.cc:1074] tablet 45fe6ec58eed40f1898d4a2f5727cfa1: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move scheduled
I20240503 02:13:22.275372 10315 rebalancer_tool.cc:1084] tablet ceae1c7c45a34b24badbbe23cb9e0176: 'd986fcfb926e47baa7d2d5b99f697334' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move ignored: Not found: Peer with uuid d986fcfb926e47baa7d2d5b99f697334 not found in consensus config
I20240503 02:13:22.275532 10315 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:25.578019 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:25.578352 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:25.578570 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:25.578787 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:25.578924 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b1c050f6f652453f9abd0ce2412df9e9: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:25.579054 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server 03f99420df704238ab300ad6e58d878c
I20240503 02:13:26.008610 10315 rebalancer_tool.cc:1074] tablet 7e966caa651e4e42890be5d79f851a1f: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '' move scheduled
I20240503 02:13:26.302112 10315 rebalancer_tool.cc:1074] tablet 402de43677c44deab8793da1895d4fb2: '2942efc0ecb448d2ba2f6bca20fbb29c' -> '03f99420df704238ab300ad6e58d878c' move scheduled
I20240503 02:13:26.537139 10315 rebalancer_tool.cc:1074] tablet 007b59de142c4b95966494ba097b1a09: 'd986fcfb926e47baa7d2d5b99f697334' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move scheduled
I20240503 02:13:26.751408 10315 rebalancer_tool.cc:1074] tablet 6043da7d2e584af885ff6a5bf8f06912: '8d7c037d50c241b484ee843b06313a64' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move scheduled
I20240503 02:13:27.151266 10315 rebalancer_tool.cc:1074] tablet 4f2cedba9f9a4e709623fceeffcfd095: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move scheduled
I20240503 02:13:27.499064 10315 rebalancer_tool.cc:1074] tablet 72e2bb75da654e8ab45bba8ac077f28c: 'd986fcfb926e47baa7d2d5b99f697334' -> '' move scheduled
I20240503 02:13:27.801522 10315 rebalancer_tool.cc:1074] tablet 80b4e6aa20f64473be8bcbf47baf66e4: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '03f99420df704238ab300ad6e58d878c' move scheduled
I20240503 02:13:28.202916 10315 rebalancer_tool.cc:1074] tablet 2d69c71d9dfd47a1a2429e7caf5e33ac: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move scheduled
I20240503 02:13:28.467705 10315 rebalancer_tool.cc:1074] tablet 5ea56405515a40f785726bd58ec3d26a: '8d7c037d50c241b484ee843b06313a64' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move scheduled
I20240503 02:13:28.856125 10315 rebalancer_tool.cc:1074] tablet 8a8affdde4d8417f95f0585dd01e7d59: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move scheduled
I20240503 02:13:29.391355 10315 rebalancer_tool.cc:1074] tablet d3d4b912095244de816d3089a31b8649: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move scheduled
I20240503 02:13:29.468354 10315 rebalancer_tool.cc:1084] tablet 4af5884849ec4ac3891ebf0f08ee2cff: 'd986fcfb926e47baa7d2d5b99f697334' -> '' move ignored: Not found: Peer with uuid d986fcfb926e47baa7d2d5b99f697334 not found in consensus config
I20240503 02:13:29.468508 10315 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:30.857084 12673 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.0.145.62:36865, user_credentials={real_user=slave}} blocked reactor thread for 48111.6us
W20240503 02:13:32.221035 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server 03f99420df704238ab300ad6e58d878c to server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d
W20240503 02:13:32.221222 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d
W20240503 02:13:32.221324 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8da59f26450d4d25a86bc61be2943c37: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d986fcfb926e47baa7d2d5b99f697334
W20240503 02:13:32.221522 10315 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:32.815240 10315 rebalancer_tool.cc:1074] tablet b64f66ace86e42f3ac213ba85bcab584: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '' move scheduled
I20240503 02:13:32.987473 10315 rebalancer_tool.cc:1074] tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298: '8d7c037d50c241b484ee843b06313a64' -> '' move scheduled
I20240503 02:13:33.458508 10315 rebalancer_tool.cc:1074] tablet cab70b62bdc747a9b653bffde680f37e: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '8d7c037d50c241b484ee843b06313a64' move scheduled
I20240503 02:13:33.710142 10315 rebalancer_tool.cc:1136] tablet cab70b62bdc747a9b653bffde680f37e: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '8d7c037d50c241b484ee843b06313a64' move completed: Incomplete: tablet cab70b62bdc747a9b653bffde680f37e, TS 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> TS 8d7c037d50c241b484ee843b06313a64 move failed, target replica disappeared
I20240503 02:13:34.004825 10315 rebalancer_tool.cc:1136] tablet d3d4b912095244de816d3089a31b8649: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move completed: OK
I20240503 02:13:35.011072 10315 rebalancer_tool.cc:1136] tablet 72e2bb75da654e8ab45bba8ac077f28c: 'd986fcfb926e47baa7d2d5b99f697334' -> '' move completed: OK
I20240503 02:13:35.050174 10315 rebalancer_tool.cc:1136] tablet 4f2cedba9f9a4e709623fceeffcfd095: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move completed: OK
I20240503 02:13:35.210374 10315 rebalancer_tool.cc:1136] tablet 80b4e6aa20f64473be8bcbf47baf66e4: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '03f99420df704238ab300ad6e58d878c' move completed: OK
I20240503 02:13:35.268203 10315 rebalancer_tool.cc:1136] tablet 7e966caa651e4e42890be5d79f851a1f: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '' move completed: OK
I20240503 02:13:35.491044 10315 rebalancer_tool.cc:1123] tablet 45fe6ec58eed40f1898d4a2f5727cfa1: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move is abandoned: Illegal state: 45fe6ec58eed40f1898d4a2f5727cfa1: source replica 2942efc0ecb448d2ba2f6bca20fbb29c does not have REPLACE attribute set
I20240503 02:13:35.839520 10315 rebalancer_tool.cc:1136] tablet 5ea56405515a40f785726bd58ec3d26a: '8d7c037d50c241b484ee843b06313a64' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move completed: OK
I20240503 02:13:36.074831 10315 rebalancer_tool.cc:1136] tablet 402de43677c44deab8793da1895d4fb2: '2942efc0ecb448d2ba2f6bca20fbb29c' -> '03f99420df704238ab300ad6e58d878c' move completed: Incomplete: tablet 402de43677c44deab8793da1895d4fb2, TS 2942efc0ecb448d2ba2f6bca20fbb29c -> TS 03f99420df704238ab300ad6e58d878c move failed, target replica disappeared
I20240503 02:13:36.209729 10315 rebalancer_tool.cc:1136] tablet 8a8affdde4d8417f95f0585dd01e7d59: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move completed: OK
I20240503 02:13:36.564697 10315 rebalancer_tool.cc:1136] tablet 007b59de142c4b95966494ba097b1a09: 'd986fcfb926e47baa7d2d5b99f697334' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move completed: OK
I20240503 02:13:36.746966 10315 rebalancer_tool.cc:1136] tablet 6043da7d2e584af885ff6a5bf8f06912: '8d7c037d50c241b484ee843b06313a64' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move completed: OK
I20240503 02:13:36.978655 10315 rebalancer_tool.cc:1074] tablet c23b06764ce54acb8388de94aea4cd4b: '8d7c037d50c241b484ee843b06313a64' -> '' move scheduled
I20240503 02:13:37.458209 10315 rebalancer_tool.cc:1136] tablet 2d69c71d9dfd47a1a2429e7caf5e33ac: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move completed: OK
I20240503 02:13:37.511371 10315 rebalancer_tool.cc:1136] tablet 5ba1f505010b403c91d3870cee19952a: '8d7c037d50c241b484ee843b06313a64' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move completed: OK
I20240503 02:18:36.129268 10315 rebalancer_tool.cc:1136] tablet 7e4acc8b5ceb4cd5b0f3708ff4f19298: '8d7c037d50c241b484ee843b06313a64' -> '' move completed: OK
W20240503 02:18:36.797344 27553 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:18:36.776216 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.145.8:38885 (local address 127.0.0.1:54608)
0503 02:18:36.776754 (+   538us) negotiation.cc:107] Waiting for socket to connect
0503 02:18:36.776776 (+    22us) client_negotiation.cc:174] Beginning negotiation
0503 02:18:36.776914 (+   138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:18:36.796004 (+ 19090us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":277,"threads_started":1}
I20240503 02:18:36.798028 10315 rebalancer_tool.cc:1123] tablet b64f66ace86e42f3ac213ba85bcab584: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:18:36.798166 10315 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): unacceptable health status UNAVAILABLE

I20240503 02:18:37.715663 27659 tablet_copy_client.cc:308] T 2448e12efbd44a90b1f8bce9519e73e0 P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38203
I20240503 02:18:37.716496  8252 tablet_copy_service.cc:140] P 2942efc0ecb448d2ba2f6bca20fbb29c: Received BeginTabletCopySession request for tablet 2448e12efbd44a90b1f8bce9519e73e0 from peer d986fcfb926e47baa7d2d5b99f697334 ({username='slave'} at 127.0.145.1:39931)
I20240503 02:18:37.716678  8252 tablet_copy_service.cc:161] P 2942efc0ecb448d2ba2f6bca20fbb29c: Beginning new tablet copy session on tablet 2448e12efbd44a90b1f8bce9519e73e0 from peer d986fcfb926e47baa7d2d5b99f697334 at {username='slave'} at 127.0.145.1:39931: session id = d986fcfb926e47baa7d2d5b99f697334-2448e12efbd44a90b1f8bce9519e73e0
I20240503 02:18:37.719045  8252 tablet_copy_source_session.cc:205] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:18:37.720501 27659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2448e12efbd44a90b1f8bce9519e73e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:37.725122 27659 tablet_copy_client.cc:783] T 2448e12efbd44a90b1f8bce9519e73e0 P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Starting download of 5 data blocks...
I20240503 02:18:37.732105 27659 tablet_copy_client.cc:647] T 2448e12efbd44a90b1f8bce9519e73e0 P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:37.734004 27659 tablet_copy_client.cc:523] T 2448e12efbd44a90b1f8bce9519e73e0 P d986fcfb926e47baa7d2d5b99f697334: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:18:37.736850 27659 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap starting.
I20240503 02:18:37.760074 27659 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:37.760578 27659 tablet_bootstrap.cc:492] T 2448e12efbd44a90b1f8bce9519e73e0 P d986fcfb926e47baa7d2d5b99f697334: Bootstrap complete.
I20240503 02:18:37.760923 27659 ts_tablet_manager.cc:1367] T 2448e12efbd44a90b1f8bce9519e73e0 P d986fcfb926e47baa7d2d5b99f697334: Time spent bootstrapping tablet: real 0.024s	user 0.024s	sys 0.000s
I20240503 02:18:37.761847 27040 raft_consensus.cc:482] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca05c9a8bee148dab8a2a982b8fa2ea4)
I20240503 02:18:37.762032 27040 raft_consensus.cc:504] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:37.761936 27659 raft_consensus.cc:348] T 2448e12efbd44a90b1f8bce9519e73e0 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } }
I20240503 02:18:37.762276 27659 raft_consensus.cc:729] T 2448e12efbd44a90b1f8bce9519e73e0 P d986fcfb926e47baa7d2d5b99f697334 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d986fcfb926e47baa7d2d5b99f697334, State: Initialized, Role: LEARNER
I20240503 02:18:37.762569 27659 consensus_queue.cc:260] T 2448e12efbd44a90b1f8bce9519e73e0 P d986fcfb926e47baa7d2d5b99f697334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } attrs { replace: true } } peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { promote: false } } peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: NON_VOTER last_known_addr { host: "127.0.145.1" port: 37681 } attrs { promote: true } }
I20240503 02:18:37.762892 27040 leader_election.cc:288] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203), ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
I20240503 02:18:37.763322 27659 ts_tablet_manager.cc:1398] T 2448e12efbd44a90b1f8bce9519e73e0 P d986fcfb926e47baa7d2d5b99f697334: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 02:18:37.764794 10310 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240503 02:13:13.662477 10318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:13:13.663053 10318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:13:13.675376 10318 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:13:16.969938 10318 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:13:19.751935 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:19.754048 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7d72eaa895849fcb02235c1b6e8d08f: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:19.754719 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:19.757166 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:19.757673 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:19.759788 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:19.760205 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:19.760481 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:19.761197 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:19.761550 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5f6139f12f9b4ab18ce35a8831b39150: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:19.761821 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7d72eaa895849fcb02235c1b6e8d08f: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:19.762297 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8da59f26450d4d25a86bc61be2943c37: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:19.764518 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 938b875abcf9426ead26522c081d6b7c: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server ca05c9a8bee148dab8a2a982b8fa2ea4
W20240503 02:13:19.765734 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:19.766063 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d4270f8900eb45459ee1e49c30ff6cff: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:19.768368 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c7d72eaa895849fcb02235c1b6e8d08f: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:20.112078 10318 rebalancer_tool.cc:1074] tablet 0f44b2b53de84a87a8757c0d00bfed37: '8d7c037d50c241b484ee843b06313a64' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move scheduled
I20240503 02:13:20.323114 10318 rebalancer_tool.cc:1074] tablet f14607425da84bb4b162a5789daa844c: 'd986fcfb926e47baa7d2d5b99f697334' -> '03f99420df704238ab300ad6e58d878c' move scheduled
I20240503 02:13:21.250983 10318 rebalancer_tool.cc:1074] tablet 72e2bb75da654e8ab45bba8ac077f28c: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'd826d4dae2aa4e05b2f64c9b17a1126f' move scheduled
I20240503 02:13:21.677201 10318 rebalancer_tool.cc:1074] tablet c45c5a14b85447c78fe2a27be4392c88: '8d7c037d50c241b484ee843b06313a64' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move scheduled
I20240503 02:13:21.801105 10318 rebalancer_tool.cc:1084] tablet aeaee03abc644807b8278cfb78e4f02b: '2942efc0ecb448d2ba2f6bca20fbb29c' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move ignored: Not found: Peer with uuid 2942efc0ecb448d2ba2f6bca20fbb29c not found in consensus config
I20240503 02:13:21.801275 10318 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:24.550385 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:24.550733 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:24.550873 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:24.680696 10318 rebalancer_tool.cc:1084] tablet de9c2a3396864696af88be1ca12ee0f1: '2942efc0ecb448d2ba2f6bca20fbb29c' -> '' move ignored: Not found: Peer with uuid 2942efc0ecb448d2ba2f6bca20fbb29c not found in consensus config
I20240503 02:13:24.680888 10318 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:26.865250 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:26.865417 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server d986fcfb926e47baa7d2d5b99f697334 to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:26.865546 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server d826d4dae2aa4e05b2f64c9b17a1126f
W20240503 02:13:26.865621 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:26.865799 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server b85e85e631ab4e04a86bfe5a9f0aa512
I20240503 02:13:26.963788 10318 rebalancer_tool.cc:1074] tablet c23b06764ce54acb8388de94aea4cd4b: '2942efc0ecb448d2ba2f6bca20fbb29c' -> '' move scheduled
I20240503 02:13:27.298863 10318 rebalancer_tool.cc:1084] tablet 4f2cedba9f9a4e709623fceeffcfd095: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move ignored: Illegal state: tablet 4f2cedba9f9a4e709623fceeffcfd095: replica b85e85e631ab4e04a86bfe5a9f0aa512 is already present
I20240503 02:13:27.299003 10318 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:30.530843 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:30.531345 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9af1e1dd28c144d9b52b73387f91d232: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:30.531951 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e33f513f068647b9921daa3a38cb3c30: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server b85e85e631ab4e04a86bfe5a9f0aa512
W20240503 02:13:30.532337 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f2b1fe68a7e4b59a12d8521e9e85cbb: could not find any suitable replica to move from server 2942efc0ecb448d2ba2f6bca20fbb29c to server 03f99420df704238ab300ad6e58d878c
W20240503 02:13:30.532625 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 635c8841bff54466a804d93ade171b2f: could not find any suitable replica to move from server 8d7c037d50c241b484ee843b06313a64 to server ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:13:30.717147 10318 rebalancer_tool.cc:1074] tablet aeaee03abc644807b8278cfb78e4f02b: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '' move scheduled
I20240503 02:13:30.973773 10318 rebalancer_tool.cc:1084] tablet 32e527fa9b2f441cb5668435681b4ff6: '8d7c037d50c241b484ee843b06313a64' -> '03f99420df704238ab300ad6e58d878c' move ignored: Illegal state: tablet 32e527fa9b2f441cb5668435681b4ff6: replica 03f99420df704238ab300ad6e58d878c is already present
I20240503 02:13:30.973907 10318 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:13:33.865078 10318 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b1c050f6f652453f9abd0ce2412df9e9: could not find any suitable replica to move from server 9cbd8ddb1fc6479eb8cacbfd5b9bf86d to server d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:13:34.104283 10318 rebalancer_tool.cc:1074] tablet ecb5c78a22a74728ab40f1da8f90388c: 'd986fcfb926e47baa7d2d5b99f697334' -> '03f99420df704238ab300ad6e58d878c' move scheduled
I20240503 02:13:34.301698 10318 rebalancer_tool.cc:1074] tablet 45fe6ec58eed40f1898d4a2f5727cfa1: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '2942efc0ecb448d2ba2f6bca20fbb29c' move scheduled
I20240503 02:13:34.489051 10318 rebalancer_tool.cc:1074] tablet 2448e12efbd44a90b1f8bce9519e73e0: '8d7c037d50c241b484ee843b06313a64' -> '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' move scheduled
I20240503 02:13:34.777076 10318 rebalancer_tool.cc:1074] tablet 4af5884849ec4ac3891ebf0f08ee2cff: '03f99420df704238ab300ad6e58d878c' -> '' move scheduled
I20240503 02:13:35.363113 10318 rebalancer_tool.cc:1074] tablet 80b4e6aa20f64473be8bcbf47baf66e4: '03f99420df704238ab300ad6e58d878c' -> '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' move scheduled
I20240503 02:13:35.988513 10318 rebalancer_tool.cc:1136] tablet f14607425da84bb4b162a5789daa844c: 'd986fcfb926e47baa7d2d5b99f697334' -> '03f99420df704238ab300ad6e58d878c' move completed: Incomplete: tablet f14607425da84bb4b162a5789daa844c, TS d986fcfb926e47baa7d2d5b99f697334 -> TS 03f99420df704238ab300ad6e58d878c move failed, target replica disappeared
I20240503 02:13:36.136996 10318 rebalancer_tool.cc:1136] tablet 0f44b2b53de84a87a8757c0d00bfed37: '8d7c037d50c241b484ee843b06313a64' -> 'b85e85e631ab4e04a86bfe5a9f0aa512' move completed: OK
I20240503 02:13:37.240823 10318 rebalancer_tool.cc:1123] tablet c45c5a14b85447c78fe2a27be4392c88: '8d7c037d50c241b484ee843b06313a64' -> 'ca05c9a8bee148dab8a2a982b8fa2ea4' move is abandoned: Illegal state: c45c5a14b85447c78fe2a27be4392c88: source replica 8d7c037d50c241b484ee843b06313a64 does not have REPLACE attribute set
I20240503 02:13:37.407055 10318 rebalancer_tool.cc:1136] tablet c23b06764ce54acb8388de94aea4cd4b: '2942efc0ecb448d2ba2f6bca20fbb29c' -> '' move completed: OK
I20240503 02:13:37.493161 10318 rebalancer_tool.cc:1136] tablet 80b4e6aa20f64473be8bcbf47baf66e4: '03f99420df704238ab300ad6e58d878c' -> '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' move completed: OK
I20240503 02:13:37.805877 10318 rebalancer_tool.cc:1136] tablet 4af5884849ec4ac3891ebf0f08ee2cff: '03f99420df704238ab300ad6e58d878c' -> '' move completed: OK
I20240503 02:13:38.940578 10318 rebalancer_tool.cc:1136] tablet 45fe6ec58eed40f1898d4a2f5727cfa1: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '2942efc0ecb448d2ba2f6bca20fbb29c' move completed: OK
I20240503 02:18:34.235860 10318 rebalancer_tool.cc:1136] tablet ecb5c78a22a74728ab40f1da8f90388c: 'd986fcfb926e47baa7d2d5b99f697334' -> '03f99420df704238ab300ad6e58d878c' move completed: Incomplete: tablet ecb5c78a22a74728ab40f1da8f90388c, TS d986fcfb926e47baa7d2d5b99f697334 -> TS 03f99420df704238ab300ad6e58d878c move failed, target replica disappeared
W20240503 02:18:36.819942 27510 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:18:36.463718 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.145.7:39415 (local address 127.0.0.1:54038)
0503 02:18:36.482892 (+ 19174us) negotiation.cc:107] Waiting for socket to connect
0503 02:18:36.482918 (+    26us) client_negotiation.cc:174] Beginning negotiation
0503 02:18:36.483074 (+   156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:18:36.818561 (+335487us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":18997,"thread_start_us":19774,"threads_started":1}
I20240503 02:18:36.821247 10318 rebalancer_tool.cc:1123] tablet aeaee03abc644807b8278cfb78e4f02b: '9cbd8ddb1fc6479eb8cacbfd5b9bf86d' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:18:36.821365 10318 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): unacceptable health status UNAVAILABLE

I20240503 02:18:37.765228  8252 tablet_copy_service.cc:342] P 2942efc0ecb448d2ba2f6bca20fbb29c: Request end of tablet copy session d986fcfb926e47baa7d2d5b99f697334-2448e12efbd44a90b1f8bce9519e73e0 received from {username='slave'} at 127.0.145.1:39931
I20240503 02:18:37.765364  8252 tablet_copy_service.cc:433] P 2942efc0ecb448d2ba2f6bca20fbb29c: ending tablet copy session d986fcfb926e47baa7d2d5b99f697334-2448e12efbd44a90b1f8bce9519e73e0 on tablet 2448e12efbd44a90b1f8bce9519e73e0 with peer d986fcfb926e47baa7d2d5b99f697334
E20240503 02:18:37.765751   580 external_mini_cluster.cc:943] Process with UUID 03f99420df704238ab300ad6e58d878c has crashed
E20240503 02:18:37.765923   580 external_mini_cluster.cc:943] Process with UUID d826d4dae2aa4e05b2f64c9b17a1126f has crashed
E20240503 02:18:37.766034   580 external_mini_cluster.cc:943] Process with UUID ca05c9a8bee148dab8a2a982b8fa2ea4 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 3
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
W20240503 02:18:37.767304  8297 leader_election.cc:334] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:37.767989   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 7889
I20240503 02:18:37.774318  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e966caa651e4e42890be5d79f851a1f" candidate_uuid: "8d7c037d50c241b484ee843b06313a64" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:18:37.779000  8295 leader_election.cc:302] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d7c037d50c241b484ee843b06313a64; no voters: 2942efc0ecb448d2ba2f6bca20fbb29c, ca05c9a8bee148dab8a2a982b8fa2ea4
I20240503 02:18:37.779420 27613 raft_consensus.cc:2726] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:18:37.784482  8011 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:18:37.815774 27579 raft_consensus.cc:482] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca05c9a8bee148dab8a2a982b8fa2ea4)
I20240503 02:18:37.815982 27579 raft_consensus.cc:504] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:37.816603 27579 leader_election.cc:288] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
I20240503 02:18:37.817338  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e966caa651e4e42890be5d79f851a1f" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64" is_pre_election: true
I20240503 02:18:37.817591  8361 raft_consensus.cc:2454] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 2.
I20240503 02:18:37.818086  8168 leader_election.cc:302] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64; no voters: 
I20240503 02:18:37.818729 27264 raft_consensus.cc:2781] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:18:37.818851 27264 raft_consensus.cc:482] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Starting leader election (detected failure of leader ca05c9a8bee148dab8a2a982b8fa2ea4)
I20240503 02:18:37.818929 27264 raft_consensus.cc:3037] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:18:37.819406  8167 leader_election.cc:334] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:37.822176 27264 raft_consensus.cc:504] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:37.823333 27264 leader_election.cc:288] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 election: Requested vote from peers 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377), ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
I20240503 02:18:37.823325  8361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e966caa651e4e42890be5d79f851a1f" candidate_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "8d7c037d50c241b484ee843b06313a64"
I20240503 02:18:37.823539  8361 raft_consensus.cc:3037] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:18:37.825385  8361 raft_consensus.cc:2454] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2942efc0ecb448d2ba2f6bca20fbb29c in term 3.
I20240503 02:18:37.826002  8168 leader_election.cc:302] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 8d7c037d50c241b484ee843b06313a64; no voters: 
I20240503 02:18:37.826459 27264 raft_consensus.cc:2781] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 3 FOLLOWER]: Leader election won for term 3
W20240503 02:18:37.826435  8167 leader_election.cc:334] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:37.826682 27264 raft_consensus.cc:686] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [term 3 LEADER]: Becoming Leader. State: Replica: 2942efc0ecb448d2ba2f6bca20fbb29c, State: Running, Role: LEADER
I20240503 02:18:37.826948 27264 consensus_queue.cc:237] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } }
I20240503 02:18:37.830108  7830 catalog_manager.cc:5484] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c reported cstate change: term changed from 2 to 3, leader changed from ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8) to 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3). New cstate: current_term: 3 leader_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: NON_VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:37.937268 27643 raft_consensus.cc:482] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d826d4dae2aa4e05b2f64c9b17a1126f)
I20240503 02:18:37.937494 27643 raft_consensus.cc:504] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:37.938174 27643 leader_election.cc:288] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:18:37.939014  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" is_pre_election: true
I20240503 02:18:37.939245  8232 raft_consensus.cc:2454] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 2.
W20240503 02:18:37.939647  8035 leader_election.cc:334] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111)
I20240503 02:18:37.939870  8035 leader_election.cc:302] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:18:37.940114 27643 raft_consensus.cc:2781] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:18:37.940207 27643 raft_consensus.cc:482] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Starting leader election (detected failure of leader d826d4dae2aa4e05b2f64c9b17a1126f)
I20240503 02:18:37.940295 27643 raft_consensus.cc:3037] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:18:37.942202 27643 raft_consensus.cc:504] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:37.942761 27643 leader_election.cc:288] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 election: Requested vote from peers d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
I20240503 02:18:37.943488  8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4acc8b5ceb4cd5b0f3708ff4f19298" candidate_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c"
I20240503 02:18:37.943679  8232 raft_consensus.cc:3037] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:18:37.943955  8035 leader_election.cc:334] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111)
I20240503 02:18:37.946627  8232 raft_consensus.cc:2454] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cbd8ddb1fc6479eb8cacbfd5b9bf86d in term 3.
I20240503 02:18:37.947074  8035 leader_election.cc:302] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2942efc0ecb448d2ba2f6bca20fbb29c, 9cbd8ddb1fc6479eb8cacbfd5b9bf86d; no voters: d826d4dae2aa4e05b2f64c9b17a1126f
I20240503 02:18:37.947389 27643 raft_consensus.cc:2781] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:18:37.947794 27643 raft_consensus.cc:686] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 LEADER]: Becoming Leader. State: Replica: 9cbd8ddb1fc6479eb8cacbfd5b9bf86d, State: Running, Role: LEADER
I20240503 02:18:37.948326 27643 consensus_queue.cc:237] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:37.951486  7830 catalog_manager.cc:5484] T 7e4acc8b5ceb4cd5b0f3708ff4f19298 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d reported cstate change: term changed from 2 to 3, leader changed from d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7) to 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2). New cstate: current_term: 3 leader_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:18:37.957288  9931 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.037544  8295 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:38.077664  8101 raft_consensus.cc:1261] T b64f66ace86e42f3ac213ba85bcab584 P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d [term 3 FOLLOWER]: Refusing update from remote peer 8d7c037d50c241b484ee843b06313a64: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 02:18:38.078328 27613 consensus_queue.cc:1035] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 02:18:38.085817  8297 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:38.104564  8017 maintenance_manager.cc:419] P d986fcfb926e47baa7d2d5b99f697334: Scheduling UndoDeltaBlockGCOp(2448e12efbd44a90b1f8bce9519e73e0): 4581 bytes on disk
I20240503 02:18:38.105286  7908 maintenance_manager.cc:643] P d986fcfb926e47baa7d2d5b99f697334: UndoDeltaBlockGCOp(2448e12efbd44a90b1f8bce9519e73e0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 02:18:38.106302   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 7889
I20240503 02:18:38.139642  8232 raft_consensus.cc:1261] T 0f44b2b53de84a87a8757c0d00bfed37 P 2942efc0ecb448d2ba2f6bca20fbb29c [term 3 FOLLOWER]: Refusing update from remote peer b85e85e631ab4e04a86bfe5a9f0aa512: Log matching property violated. Preceding OpId in replica: term: 2 index: 18. Preceding OpId from leader: term: 3 index: 19. (index mismatch)
I20240503 02:18:38.140209 27372 consensus_queue.cc:1035] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20240503 02:18:38.168808  8168 connection.cc:535] server connection from 127.0.145.1:39931 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:18:38.169833   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 8020
W20240503 02:18:38.170588  8035 consensus_peers.cc:485] T 2d69c71d9dfd47a1a2429e7caf5e33ac P 9cbd8ddb1fc6479eb8cacbfd5b9bf86d -> Peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Couldn't send request to peer d986fcfb926e47baa7d2d5b99f697334. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:37681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.171018  9929 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Couldn't send request to peer d986fcfb926e47baa7d2d5b99f697334. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:37681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.172186  8295 consensus_peers.cc:485] T 834aaba965c8443c862e98a070e36cac P 8d7c037d50c241b484ee843b06313a64 -> Peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Couldn't send request to peer d986fcfb926e47baa7d2d5b99f697334. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:37681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.176316  9931 consensus_peers.cc:462] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:38.178221  8140 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:18:38.210019  8167 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.366430  8165 consensus_peers.cc:485] T 6043da7d2e584af885ff6a5bf8f06912 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Couldn't send request to peer d986fcfb926e47baa7d2d5b99f697334. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.1:37681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.425710  8165 consensus_peers.cc:485] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:38.442193  8361 raft_consensus.cc:1261] T 7e966caa651e4e42890be5d79f851a1f P 8d7c037d50c241b484ee843b06313a64 [term 3 FOLLOWER]: Refusing update from remote peer 2942efc0ecb448d2ba2f6bca20fbb29c: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 3 index: 16. (index mismatch)
I20240503 02:18:38.442801 27685 consensus_queue.cc:1035] T 7e966caa651e4e42890be5d79f851a1f P 2942efc0ecb448d2ba2f6bca20fbb29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240503 02:18:38.464989  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2448e12efbd44a90b1f8bce9519e73e0: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:18:38.517385   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 8020
W20240503 02:18:38.579738  8297 connection.cc:535] client connection to 127.0.145.2:35309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:38.579833  8167 connection.cc:535] client connection to 127.0.145.2:35309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:18:38.580118   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 8150
W20240503 02:18:38.580963  8297 consensus_peers.cc:485] T de9c2a3396864696af88be1ca12ee0f1 P 8d7c037d50c241b484ee843b06313a64 -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:35309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.581017  8167 consensus_peers.cc:485] T ceae1c7c45a34b24badbbe23cb9e0176 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:35309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.581118  8167 consensus_peers.cc:485] T 2448e12efbd44a90b1f8bce9519e73e0 P 2942efc0ecb448d2ba2f6bca20fbb29c -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:35309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:38.586120  8270 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:18:38.659272  9931 consensus_peers.cc:462] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
W20240503 02:18:38.698714  8297 consensus_peers.cc:485] T b64f66ace86e42f3ac213ba85bcab584 P 8d7c037d50c241b484ee843b06313a64 -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:35309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.705195  9931 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:35309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.714738  8297 consensus_peers.cc:485] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:35309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:38.812536  8297 consensus_peers.cc:485] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 -> Peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309): Couldn't send request to peer 9cbd8ddb1fc6479eb8cacbfd5b9bf86d. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.2:35309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:38.821333  7819 maintenance_manager.cc:419] P 5680e1b3edfe402b831ab3889e06b093: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 11218 bytes on disk
I20240503 02:18:38.826810  7818 maintenance_manager.cc:643] P 5680e1b3edfe402b831ab3889e06b093: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s	user 0.005s	sys 0.000s Metrics: {"dirs.queue_time_us":402,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":91,"thread_start_us":283,"threads_started":1}
W20240503 02:18:38.857219  8297 consensus_peers.cc:485] T 3dbfc7f12fc142f98b1d1c026fd49f5a P 8d7c037d50c241b484ee843b06313a64 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Couldn't send request to peer ca05c9a8bee148dab8a2a982b8fa2ea4. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:18:38.942924  8295 consensus_peers.cc:485] T 402de43677c44deab8793da1895d4fb2 P 8d7c037d50c241b484ee843b06313a64 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:18:38.952539   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 8150
I20240503 02:18:39.008888   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 8280
W20240503 02:18:39.009984  9929 consensus_peers.cc:485] T 0f44b2b53de84a87a8757c0d00bfed37 P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): Couldn't send request to peer 2942efc0ecb448d2ba2f6bca20fbb29c. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.3:38203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:39.015020  8401 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:18:39.029681  9929 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Couldn't send request to peer d826d4dae2aa4e05b2f64c9b17a1126f. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:18:39.103854  9931 consensus_peers.cc:462] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.145.8:38885: connect: Connection refused (error 111)
I20240503 02:18:39.205210 27372 raft_consensus.cc:482] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d986fcfb926e47baa7d2d5b99f697334)
I20240503 02:18:39.205422 27372 raft_consensus.cc:504] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "2942efc0ecb448d2ba2f6bca20fbb29c" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38203 } attrs { promote: false } }
I20240503 02:18:39.206076 27372 leader_election.cc:288] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203)
W20240503 02:18:39.207468  9929 leader_election.cc:334] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Network error: Client connection negotiation failed: client connection to 127.0.145.1:37681: connect: Connection refused (error 111)
W20240503 02:18:39.207659  9929 leader_election.cc:334] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2942efc0ecb448d2ba2f6bca20fbb29c (127.0.145.3:38203): Network error: Client connection negotiation failed: client connection to 127.0.145.3:38203: connect: Connection refused (error 111)
I20240503 02:18:39.207759  9929 leader_election.cc:302] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b85e85e631ab4e04a86bfe5a9f0aa512; no voters: 2942efc0ecb448d2ba2f6bca20fbb29c, d986fcfb926e47baa7d2d5b99f697334
I20240503 02:18:39.208024 27372 raft_consensus.cc:2726] T 45fe6ec58eed40f1898d4a2f5727cfa1 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:18:39.272837 27372 raft_consensus.cc:482] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d986fcfb926e47baa7d2d5b99f697334)
I20240503 02:18:39.273059 27372 raft_consensus.cc:504] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } }
I20240503 02:18:39.273707 27372 leader_election.cc:288] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
W20240503 02:18:39.275278  9929 leader_election.cc:334] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681): Network error: Client connection negotiation failed: client connection to 127.0.145.1:37681: connect: Connection refused (error 111)
W20240503 02:18:39.275457  9929 leader_election.cc:334] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415): Network error: Client connection negotiation failed: client connection to 127.0.145.7:39415: connect: Connection refused (error 111)
I20240503 02:18:39.275555  9929 leader_election.cc:302] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b85e85e631ab4e04a86bfe5a9f0aa512; no voters: d826d4dae2aa4e05b2f64c9b17a1126f, d986fcfb926e47baa7d2d5b99f697334
I20240503 02:18:39.275791 27372 raft_consensus.cc:2726] T 4f2cedba9f9a4e709623fceeffcfd095 P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:18:39.292263  7817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111) [suppressed 97 similar messages]
I20240503 02:18:39.318416 27675 raft_consensus.cc:482] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:18:39.318646 27675 raft_consensus.cc:504] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cbd8ddb1fc6479eb8cacbfd5b9bf86d" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35309 } attrs { replace: true } } peers { permanent_uuid: "8d7c037d50c241b484ee843b06313a64" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 35377 } } peers { permanent_uuid: "d826d4dae2aa4e05b2f64c9b17a1126f" member_type: VOTER last_known_addr { host: "127.0.145.7" port: 39415 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: NON_VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: true } }
I20240503 02:18:39.319324 27675 leader_election.cc:288] T aeaee03abc644807b8278cfb78e4f02b P 8d7c037d50c241b484ee843b06313a64 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9cbd8ddb1fc6479eb8cacbfd5b9bf86d (127.0.145.2:35309), d826d4dae2aa4e05b2f64c9b17a1126f (127.0.145.7:39415)
I20240503 02:18:39.320792   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 8280
W20240503 02:18:39.375845  9930 connection.cc:535] server connection from 127.0.145.4:52991 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:39.375880  9932 connection.cc:535] client connection to 127.0.145.4:35377 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:18:39.376173  7815 connection.cc:535] server connection from 127.0.145.4:49523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:18:39.376392   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 9913
W20240503 02:18:39.377455  9932 consensus_peers.cc:485] T 72e2bb75da654e8ab45bba8ac077f28c P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): Couldn't send request to peer 8d7c037d50c241b484ee843b06313a64. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.4:35377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:39.377563  9932 consensus_peers.cc:485] T c23b06764ce54acb8388de94aea4cd4b P b85e85e631ab4e04a86bfe5a9f0aa512 -> Peer 8d7c037d50c241b484ee843b06313a64 (127.0.145.4:35377): Couldn't send request to peer 8d7c037d50c241b484ee843b06313a64. Status: Network error: Client connection negotiation failed: client connection to 127.0.145.4:35377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:39.382402 10035 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:18:39.522897  7817 catalog_manager.cc:4573] TS 03f99420df704238ab300ad6e58d878c (127.0.145.5:44891): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2448e12efbd44a90b1f8bce9519e73e0: Network error: Client connection negotiation failed: client connection to 127.0.145.5:44891: connect: Connection refused (error 111)
I20240503 02:18:39.708173 27680 raft_consensus.cc:482] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d986fcfb926e47baa7d2d5b99f697334)
I20240503 02:18:39.708392 27680 raft_consensus.cc:504] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:18:39.710237 27680 leader_election.cc:288] T cab70b62bdc747a9b653bffde680f37e P b85e85e631ab4e04a86bfe5a9f0aa512 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d986fcfb926e47baa7d2d5b99f697334 (127.0.145.1:37681), ca05c9a8bee148dab8a2a982b8fa2ea4 (127.0.145.8:38885)
I20240503 02:18:39.715214 27711 raft_consensus.cc:482] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d986fcfb926e47baa7d2d5b99f697334)
I20240503 02:18:39.715607   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 9913
I20240503 02:18:39.715416 27711 raft_consensus.cc:504] T 5ea56405515a40f785726bd58ec3d26a P b85e85e631ab4e04a86bfe5a9f0aa512 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d986fcfb926e47baa7d2d5b99f697334" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 37681 } } peers { permanent_uuid: "b85e85e631ab4e04a86bfe5a9f0aa512" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 40397 } attrs { promote: false } } peers { permanent_uuid: "ca05c9a8bee148dab8a2a982b8fa2ea4" member_type: VOTER last_known_addr { host: "127.0.145.8" port: 38885 } attrs { promote: false } }
I20240503 02:18:39.766579   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 7799
I20240503 02:18:39.777695  7859 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:18:40.038266   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 7799
2024-05-03T02:18:40Z chronyd exiting
I20240503 02:18:40.108736   580 test_util.cc:175] -----------------------------------------------
I20240503 02:18:40.108851   580 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714702277455189-580-0
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00> (349597 ms)
[----------] 1 test from ConcurrentRebalancersTest (349597 ms total)

[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN      ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1
I20240503 02:18:40.111020   580 test_util.cc:268] Using random seed: 1451692438
I20240503 02:18:40.112797   580 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:18:40.112888   580 ts_itest-base.cc:113] --------------
I20240503 02:18:40.112949   580 ts_itest-base.cc:114] 4 tablet servers
I20240503 02:18:40.113011   580 ts_itest-base.cc:115] 1 replicas per TS
I20240503 02:18:40.113075   580 ts_itest-base.cc:116] --------------
2024-05-03T02:18:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:18:40Z Disabled control of system clock
I20240503 02:18:40.159739   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.145.62:43077
--webserver_interface=127.0.145.62
--webserver_port=0
--builtin_ntp_servers=127.0.145.20:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.145.62:43077
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240503 02:18:40.503050 27723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:18:40.503350 27723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:18:40.503438 27723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:18:40.510811 27723 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:18:40.510939 27723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:18:40.510993 27723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:18:40.511055 27723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:18:40.511101 27723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:18:40.520265 27723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.145.62:43077
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.145.62:43077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.145.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:18:40.521618 27723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:18:40.523451 27723 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:18:40.534900 27728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:40.535313 27729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:40.535794 27731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:18:40.536581 27723 server_base.cc:1034] running on GCE node
I20240503 02:18:40.537345 27723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:18:40.538801 27723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:18:40.540007 27723 hybrid_clock.cc:648] HybridClock initialized: now 1714702720539962 us; error 38 us; skew 500 ppm
I20240503 02:18:40.540341 27723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:18:40.542572 27723 webserver.cc:460] Webserver started at http://127.0.145.62:46833/ using document root <none> and password file <none>
I20240503 02:18:40.543329 27723 fs_manager.cc:359] Metadata directory not provided
I20240503 02:18:40.543444 27723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:18:40.543761 27723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:18:40.546309 27723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1e862dc3b1c34c47b24dd8f646cb6f8e"
format_stamp: "Formatted at 2024-05-03 02:18:40 on dist-test-slave-gq9h"
I20240503 02:18:40.547078 27723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1e862dc3b1c34c47b24dd8f646cb6f8e"
format_stamp: "Formatted at 2024-05-03 02:18:40 on dist-test-slave-gq9h"
I20240503 02:18:40.552183 27723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:18:40.555642 27737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:40.556977 27723 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:18:40.557163 27723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1e862dc3b1c34c47b24dd8f646cb6f8e"
format_stamp: "Formatted at 2024-05-03 02:18:40 on dist-test-slave-gq9h"
I20240503 02:18:40.557381 27723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:18:40.579054 27723 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:18:40.579931 27723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:18:40.580214 27723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:18:40.598052 27723 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.62:43077
I20240503 02:18:40.598066 27788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.62:43077 every 8 connection(s)
I20240503 02:18:40.599519 27723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:18:40.603294 27789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:40.608703   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 27723
I20240503 02:18:40.608919   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:18:40.613766 27789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e: Bootstrap starting.
I20240503 02:18:40.617380 27789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:40.618743 27789 log.cc:824] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e: Log is configured to *not* fsync() on all Append() calls
I20240503 02:18:40.622797 27789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e: No bootstrap required, opened a new log
I20240503 02:18:40.628676 27789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e862dc3b1c34c47b24dd8f646cb6f8e" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 43077 } }
I20240503 02:18:40.629034 27789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:40.629153 27789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e862dc3b1c34c47b24dd8f646cb6f8e, State: Initialized, Role: FOLLOWER
I20240503 02:18:40.629715 27789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e862dc3b1c34c47b24dd8f646cb6f8e" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 43077 } }
I20240503 02:18:40.629911 27789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:40.630035 27789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:40.630178 27789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:40.632306 27789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e862dc3b1c34c47b24dd8f646cb6f8e" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 43077 } }
I20240503 02:18:40.632794 27789 leader_election.cc:302] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e862dc3b1c34c47b24dd8f646cb6f8e; no voters: 
I20240503 02:18:40.633347 27789 leader_election.cc:288] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:40.633461 27794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:40.634110 27794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [term 1 LEADER]: Becoming Leader. State: Replica: 1e862dc3b1c34c47b24dd8f646cb6f8e, State: Running, Role: LEADER
I20240503 02:18:40.634693 27794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e862dc3b1c34c47b24dd8f646cb6f8e" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 43077 } }
I20240503 02:18:40.635430 27789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:18:40.637774 27795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e862dc3b1c34c47b24dd8f646cb6f8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e862dc3b1c34c47b24dd8f646cb6f8e" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 43077 } } }
I20240503 02:18:40.638240 27795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [sys.catalog]: This master's current role is: LEADER
I20240503 02:18:40.637768 27796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e862dc3b1c34c47b24dd8f646cb6f8e. Latest consensus state: current_term: 1 leader_uuid: "1e862dc3b1c34c47b24dd8f646cb6f8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e862dc3b1c34c47b24dd8f646cb6f8e" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 43077 } } }
I20240503 02:18:40.643483 27796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e [sys.catalog]: This master's current role is: LEADER
I20240503 02:18:40.643937 27799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:18:40.649770 27799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:18:40.658859 27799 catalog_manager.cc:1347] Generated new cluster ID: ec7c5122632c4230b106355361275c8f
I20240503 02:18:40.658989 27799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:18:40.666807 27799 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:18:40.668253 27799 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:18:40.676534 27799 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1e862dc3b1c34c47b24dd8f646cb6f8e: Generated new TSK 0
I20240503 02:18:40.677507 27799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:18:40.695168   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.1:0
--local_ip_for_outbound_sockets=127.0.145.1
--webserver_interface=127.0.145.1
--webserver_port=0
--tserver_master_addrs=127.0.145.62:43077
--builtin_ntp_servers=127.0.145.20:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:18:41.041035 27813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:18:41.041355 27813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:18:41.041452 27813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:18:41.048947 27813 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:18:41.049078 27813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:18:41.049194 27813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.1
I20240503 02:18:41.058048 27813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.145.1
--webserver_port=0
--tserver_master_addrs=127.0.145.62:43077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.1
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:18:41.059414 27813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:18:41.061161 27813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:18:41.073338 27819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:41.074000 27818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:41.074461 27821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:18:41.075099 27813 server_base.cc:1034] running on GCE node
I20240503 02:18:41.075682 27813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:18:41.076575 27813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:18:41.077788 27813 hybrid_clock.cc:648] HybridClock initialized: now 1714702721077742 us; error 47 us; skew 500 ppm
I20240503 02:18:41.078125 27813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:18:41.080403 27813 webserver.cc:460] Webserver started at http://127.0.145.1:39295/ using document root <none> and password file <none>
I20240503 02:18:41.081120 27813 fs_manager.cc:359] Metadata directory not provided
I20240503 02:18:41.081231 27813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:18:41.081524 27813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:18:41.084224 27813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5b5ff5d736d344869048c4155e43b56a"
format_stamp: "Formatted at 2024-05-03 02:18:41 on dist-test-slave-gq9h"
I20240503 02:18:41.084995 27813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5b5ff5d736d344869048c4155e43b56a"
format_stamp: "Formatted at 2024-05-03 02:18:41 on dist-test-slave-gq9h"
I20240503 02:18:41.090153 27813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240503 02:18:41.093487 27827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:41.094825 27813 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:18:41.094972 27813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5b5ff5d736d344869048c4155e43b56a"
format_stamp: "Formatted at 2024-05-03 02:18:41 on dist-test-slave-gq9h"
I20240503 02:18:41.095167 27813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:18:41.121357 27813 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:18:41.122260 27813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:18:41.122548 27813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:18:41.123505 27813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:18:41.124946 27813 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:18:41.125051 27813 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:41.125165 27813 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:18:41.125236 27813 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:41.156652 27813 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.1:38711
I20240503 02:18:41.156713 27939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.1:38711 every 8 connection(s)
I20240503 02:18:41.157982 27813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:18:41.160773   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 27813
I20240503 02:18:41.160998   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:18:41.164115   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.2:0
--local_ip_for_outbound_sockets=127.0.145.2
--webserver_interface=127.0.145.2
--webserver_port=0
--tserver_master_addrs=127.0.145.62:43077
--builtin_ntp_servers=127.0.145.20:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:18:41.171001 27940 heartbeater.cc:347] Connected to a master server at 127.0.145.62:43077
I20240503 02:18:41.171365 27940 heartbeater.cc:464] Registering TS with master...
I20240503 02:18:41.172426 27940 heartbeater.cc:511] Master 127.0.145.62:43077 requested a full tablet report, sending...
I20240503 02:18:41.174547 27754 ts_manager.cc:194] Registered new tserver with Master: 5b5ff5d736d344869048c4155e43b56a (127.0.145.1:38711)
I20240503 02:18:41.176811 27754 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.1:58797
W20240503 02:18:41.507225 27944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:18:41.507547 27944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:18:41.507663 27944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:18:41.515584 27944 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:18:41.515733 27944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:18:41.515873 27944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.2
I20240503 02:18:41.525223 27944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.145.2
--webserver_port=0
--tserver_master_addrs=127.0.145.62:43077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.2
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:18:41.526573 27944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:18:41.528528 27944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:18:41.541440 27949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:41.541774 27950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:41.542230 27952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:18:41.543630 27944 server_base.cc:1034] running on GCE node
I20240503 02:18:41.544253 27944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:18:41.545290 27944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:18:41.546489 27944 hybrid_clock.cc:648] HybridClock initialized: now 1714702721546450 us; error 43 us; skew 500 ppm
I20240503 02:18:41.546861 27944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:18:41.549108 27944 webserver.cc:460] Webserver started at http://127.0.145.2:34065/ using document root <none> and password file <none>
I20240503 02:18:41.549837 27944 fs_manager.cc:359] Metadata directory not provided
I20240503 02:18:41.549962 27944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:18:41.550269 27944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:18:41.552909 27944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "fe00ee12f19c43b3aef56af3b569ee00"
format_stamp: "Formatted at 2024-05-03 02:18:41 on dist-test-slave-gq9h"
I20240503 02:18:41.553680 27944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "fe00ee12f19c43b3aef56af3b569ee00"
format_stamp: "Formatted at 2024-05-03 02:18:41 on dist-test-slave-gq9h"
I20240503 02:18:41.559046 27944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240503 02:18:41.562501 27958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:41.563856 27944 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:18:41.564023 27944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fe00ee12f19c43b3aef56af3b569ee00"
format_stamp: "Formatted at 2024-05-03 02:18:41 on dist-test-slave-gq9h"
I20240503 02:18:41.564235 27944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:18:41.577282 27944 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:18:41.578119 27944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:18:41.578385 27944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:18:41.579455 27944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:18:41.580929 27944 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:18:41.581028 27944 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:41.581153 27944 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:18:41.581228 27944 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:41.615693 27944 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.2:35911
I20240503 02:18:41.615736 28070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.2:35911 every 8 connection(s)
I20240503 02:18:41.617012 27944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:18:41.626803   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 27944
I20240503 02:18:41.627004   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:18:41.630510   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.3:0
--local_ip_for_outbound_sockets=127.0.145.3
--webserver_interface=127.0.145.3
--webserver_port=0
--tserver_master_addrs=127.0.145.62:43077
--builtin_ntp_servers=127.0.145.20:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:18:41.630695 28071 heartbeater.cc:347] Connected to a master server at 127.0.145.62:43077
I20240503 02:18:41.630982 28071 heartbeater.cc:464] Registering TS with master...
I20240503 02:18:41.631839 28071 heartbeater.cc:511] Master 127.0.145.62:43077 requested a full tablet report, sending...
I20240503 02:18:41.633222 27754 ts_manager.cc:194] Registered new tserver with Master: fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2:35911)
I20240503 02:18:41.633953 27754 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.2:55653
W20240503 02:18:41.962186 28074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:18:41.962491 28074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:18:41.962666 28074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:18:41.972889 28074 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:18:41.973032 28074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:18:41.973199 28074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.3
I20240503 02:18:41.982046 28074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.145.3
--webserver_port=0
--tserver_master_addrs=127.0.145.62:43077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.3
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:18:41.983443 28074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:18:41.985147 28074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:18:41.997179 28079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:41.997678 28080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:41.998111 28082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:18:41.998991 28074 server_base.cc:1034] running on GCE node
I20240503 02:18:41.999621 28074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:18:42.000622 28074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:18:42.001823 28074 hybrid_clock.cc:648] HybridClock initialized: now 1714702722001768 us; error 48 us; skew 500 ppm
I20240503 02:18:42.002127 28074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:18:42.004302 28074 webserver.cc:460] Webserver started at http://127.0.145.3:34261/ using document root <none> and password file <none>
I20240503 02:18:42.005017 28074 fs_manager.cc:359] Metadata directory not provided
I20240503 02:18:42.005125 28074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:18:42.005434 28074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:18:42.008021 28074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d85cddb1910a4ca980af31262d4474dd"
format_stamp: "Formatted at 2024-05-03 02:18:42 on dist-test-slave-gq9h"
I20240503 02:18:42.008790 28074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d85cddb1910a4ca980af31262d4474dd"
format_stamp: "Formatted at 2024-05-03 02:18:42 on dist-test-slave-gq9h"
I20240503 02:18:42.013994 28074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20240503 02:18:42.017402 28088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:42.018748 28074 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:18:42.018904 28074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d85cddb1910a4ca980af31262d4474dd"
format_stamp: "Formatted at 2024-05-03 02:18:42 on dist-test-slave-gq9h"
I20240503 02:18:42.019102 28074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:18:42.046195 28074 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:18:42.047080 28074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:18:42.047339 28074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:18:42.048275 28074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:18:42.049831 28074 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:18:42.049916 28074 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:42.050035 28074 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:18:42.050115 28074 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:42.081332 28074 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.3:36141
I20240503 02:18:42.081406 28200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.3:36141 every 8 connection(s)
I20240503 02:18:42.082765 28074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:18:42.091109   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 28074
I20240503 02:18:42.091276   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:18:42.094799   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.4:0
--local_ip_for_outbound_sockets=127.0.145.4
--webserver_interface=127.0.145.4
--webserver_port=0
--tserver_master_addrs=127.0.145.62:43077
--builtin_ntp_servers=127.0.145.20:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:18:42.095777 28201 heartbeater.cc:347] Connected to a master server at 127.0.145.62:43077
I20240503 02:18:42.096096 28201 heartbeater.cc:464] Registering TS with master...
I20240503 02:18:42.096884 28201 heartbeater.cc:511] Master 127.0.145.62:43077 requested a full tablet report, sending...
I20240503 02:18:42.098228 27754 ts_manager.cc:194] Registered new tserver with Master: d85cddb1910a4ca980af31262d4474dd (127.0.145.3:36141)
I20240503 02:18:42.099120 27754 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.3:45223
I20240503 02:18:42.180413 27940 heartbeater.cc:503] Master 127.0.145.62:43077 was elected leader, sending a full tablet report...
W20240503 02:18:42.429105 28204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:18:42.429440 28204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:18:42.429551 28204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:18:42.437278 28204 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:18:42.437409 28204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:18:42.437521 28204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.4
I20240503 02:18:42.446691 28204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.145.4
--webserver_port=0
--tserver_master_addrs=127.0.145.62:43077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.4
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:18:42.447995 28204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:18:42.449714 28204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:18:42.462044 28209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:42.462507 28210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:42.462896 28212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:18:42.463690 28204 server_base.cc:1034] running on GCE node
I20240503 02:18:42.464327 28204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:18:42.465299 28204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:18:42.466500 28204 hybrid_clock.cc:648] HybridClock initialized: now 1714702722466454 us; error 46 us; skew 500 ppm
I20240503 02:18:42.466852 28204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:18:42.469296 28204 webserver.cc:460] Webserver started at http://127.0.145.4:39907/ using document root <none> and password file <none>
I20240503 02:18:42.470053 28204 fs_manager.cc:359] Metadata directory not provided
I20240503 02:18:42.470168 28204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:18:42.470467 28204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:18:42.473150 28204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "57d37d4403774fb8b57c63a6d53f7dff"
format_stamp: "Formatted at 2024-05-03 02:18:42 on dist-test-slave-gq9h"
I20240503 02:18:42.473901 28204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "57d37d4403774fb8b57c63a6d53f7dff"
format_stamp: "Formatted at 2024-05-03 02:18:42 on dist-test-slave-gq9h"
I20240503 02:18:42.479095 28204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.008s	sys 0.000s
I20240503 02:18:42.482785 28218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:42.484148 28204 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:18:42.484309 28204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "57d37d4403774fb8b57c63a6d53f7dff"
format_stamp: "Formatted at 2024-05-03 02:18:42 on dist-test-slave-gq9h"
I20240503 02:18:42.484500 28204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:18:42.503036 28204 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:18:42.503825 28204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:18:42.504076 28204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:18:42.504963 28204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:18:42.506390 28204 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:18:42.506485 28204 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:42.506595 28204 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:18:42.506703 28204 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20240503 02:18:42.536470 28204 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.4:40035
I20240503 02:18:42.536540 28330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.4:40035 every 8 connection(s)
I20240503 02:18:42.537767 28204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:18:42.543674   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 28204
I20240503 02:18:42.543864   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:18:42.550412 28331 heartbeater.cc:347] Connected to a master server at 127.0.145.62:43077
I20240503 02:18:42.550711 28331 heartbeater.cc:464] Registering TS with master...
I20240503 02:18:42.551499 28331 heartbeater.cc:511] Master 127.0.145.62:43077 requested a full tablet report, sending...
I20240503 02:18:42.552758 27754 ts_manager.cc:194] Registered new tserver with Master: 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4:40035)
I20240503 02:18:42.553503 27754 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.4:50847
I20240503 02:18:42.559643   580 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 02:18:42.575282 27754 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54568:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240503 02:18:42.603045 28136 tablet_service.cc:1467] Processing CreateTablet for tablet a8c411abd9f14362ae6a4efbdca4ac10 (DEFAULT_TABLE table=TestTable [id=33586fd711b64b0cb49fdccbdf8228ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:18:42.604468 28136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8c411abd9f14362ae6a4efbdca4ac10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:42.618053 28345 tablet_bootstrap.cc:492] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd: Bootstrap starting.
I20240503 02:18:42.621543 28345 tablet_bootstrap.cc:654] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:42.622637 28345 log.cc:824] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd: Log is configured to *not* fsync() on all Append() calls
I20240503 02:18:42.625191 28345 tablet_bootstrap.cc:492] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd: No bootstrap required, opened a new log
I20240503 02:18:42.625468 28345 ts_tablet_manager.cc:1367] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.003s
I20240503 02:18:42.632236 28345 raft_consensus.cc:348] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } }
I20240503 02:18:42.632562 28345 raft_consensus.cc:374] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:42.632645 28345 raft_consensus.cc:729] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85cddb1910a4ca980af31262d4474dd, State: Initialized, Role: FOLLOWER
I20240503 02:18:42.633237 28345 consensus_queue.cc:260] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } }
I20240503 02:18:42.633421 28345 raft_consensus.cc:388] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:42.633546 28345 raft_consensus.cc:482] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:42.633689 28345 raft_consensus.cc:3037] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:42.635841 28345 raft_consensus.cc:504] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } }
I20240503 02:18:42.636381 28345 leader_election.cc:302] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d85cddb1910a4ca980af31262d4474dd; no voters: 
I20240503 02:18:42.637050 28345 leader_election.cc:288] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:42.637087 28071 heartbeater.cc:503] Master 127.0.145.62:43077 was elected leader, sending a full tablet report...
I20240503 02:18:42.637235 28347 raft_consensus.cc:2781] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:42.638204 28201 heartbeater.cc:503] Master 127.0.145.62:43077 was elected leader, sending a full tablet report...
I20240503 02:18:42.638918 28345 ts_tablet_manager.cc:1398] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd: Time spent starting tablet: real 0.013s	user 0.014s	sys 0.001s
I20240503 02:18:42.639447 28347 raft_consensus.cc:686] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 1 LEADER]: Becoming Leader. State: Replica: d85cddb1910a4ca980af31262d4474dd, State: Running, Role: LEADER
I20240503 02:18:42.639884 28347 consensus_queue.cc:237] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } }
I20240503 02:18:42.644989 27754 catalog_manager.cc:5484] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd reported cstate change: term changed from 0 to 1, leader changed from <none> to d85cddb1910a4ca980af31262d4474dd (127.0.145.3). New cstate: current_term: 1 leader_uuid: "d85cddb1910a4ca980af31262d4474dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } health_report { overall_health: HEALTHY } } }
I20240503 02:18:42.659509   580 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 02:18:42.661516   580 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5b5ff5d736d344869048c4155e43b56a to finish bootstrapping
I20240503 02:18:42.670719   580 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fe00ee12f19c43b3aef56af3b569ee00 to finish bootstrapping
I20240503 02:18:42.679734   580 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d85cddb1910a4ca980af31262d4474dd to finish bootstrapping
I20240503 02:18:42.687470   580 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 57d37d4403774fb8b57c63a6d53f7dff to finish bootstrapping
I20240503 02:18:42.696656   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 28074
I20240503 02:18:42.697645 28196 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:18:42.781486   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 28074
I20240503 02:18:42.798723   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 28204
I20240503 02:18:42.799924 28326 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:18:42.878950   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 28204
I20240503 02:18:46.647192 27754 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54568:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:18:46.666503 28006 tablet_service.cc:1467] Processing CreateTablet for tablet 117bdd53a4784124a5fc9704b0755144 (DEFAULT_TABLE table=rebalance_test_table_0 [id=dc591e3b8e6e4667b50bf68f42852d13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:18:46.666507 27874 tablet_service.cc:1467] Processing CreateTablet for tablet 6096f9bd19e749889c8e6154d88c921d (DEFAULT_TABLE table=rebalance_test_table_0 [id=dc591e3b8e6e4667b50bf68f42852d13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:18:46.667846 27874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6096f9bd19e749889c8e6154d88c921d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:46.667850 28006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 117bdd53a4784124a5fc9704b0755144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:46.666550 27875 tablet_service.cc:1467] Processing CreateTablet for tablet e833cff4caf947a986d4abb1df7ef4fb (DEFAULT_TABLE table=rebalance_test_table_0 [id=dc591e3b8e6e4667b50bf68f42852d13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:18:46.674717 27875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e833cff4caf947a986d4abb1df7ef4fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:46.679725 28367 tablet_bootstrap.cc:492] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00: Bootstrap starting.
I20240503 02:18:46.680771 28366 tablet_bootstrap.cc:492] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a: Bootstrap starting.
I20240503 02:18:46.683653 28366 tablet_bootstrap.cc:654] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:46.683652 28367 tablet_bootstrap.cc:654] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:46.684623 28366 log.cc:824] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a: Log is configured to *not* fsync() on all Append() calls
I20240503 02:18:46.684625 28367 log.cc:824] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00: Log is configured to *not* fsync() on all Append() calls
I20240503 02:18:46.686978 28366 tablet_bootstrap.cc:492] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a: No bootstrap required, opened a new log
I20240503 02:18:46.686978 28367 tablet_bootstrap.cc:492] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00: No bootstrap required, opened a new log
I20240503 02:18:46.687213 28367 ts_tablet_manager.cc:1367] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.001s
I20240503 02:18:46.687213 28366 ts_tablet_manager.cc:1367] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.006s
I20240503 02:18:46.693344 28366 raft_consensus.cc:348] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:46.693343 28367 raft_consensus.cc:348] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:46.693661 28366 raft_consensus.cc:374] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:46.693678 28367 raft_consensus.cc:374] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:46.693765 28367 raft_consensus.cc:729] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Initialized, Role: FOLLOWER
I20240503 02:18:46.693763 28366 raft_consensus.cc:729] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5ff5d736d344869048c4155e43b56a, State: Initialized, Role: FOLLOWER
I20240503 02:18:46.694324 28366 consensus_queue.cc:260] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:46.694336 28367 consensus_queue.cc:260] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:46.694506 28366 raft_consensus.cc:388] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:46.694517 28367 raft_consensus.cc:388] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:46.694684 28366 raft_consensus.cc:482] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:46.694687 28367 raft_consensus.cc:482] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:46.694845 28367 raft_consensus.cc:3037] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:46.694846 28366 raft_consensus.cc:3037] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:46.696734 28367 raft_consensus.cc:504] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:46.696744 28366 raft_consensus.cc:504] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:46.697166 28366 leader_election.cc:302] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b5ff5d736d344869048c4155e43b56a; no voters: 
I20240503 02:18:46.697166 28367 leader_election.cc:302] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe00ee12f19c43b3aef56af3b569ee00; no voters: 
I20240503 02:18:46.697806 28367 leader_election.cc:288] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:46.697829 28366 leader_election.cc:288] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:46.697945 28370 raft_consensus.cc:2781] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:46.700007 28371 raft_consensus.cc:2781] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:46.702116 28370 raft_consensus.cc:686] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Becoming Leader. State: Replica: 5b5ff5d736d344869048c4155e43b56a, State: Running, Role: LEADER
I20240503 02:18:46.702819 28370 consensus_queue.cc:237] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:46.703704 28371 raft_consensus.cc:686] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Becoming Leader. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Running, Role: LEADER
I20240503 02:18:46.704450 28371 consensus_queue.cc:237] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:46.707365 28366 ts_tablet_manager.cc:1398] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.010s
I20240503 02:18:46.707789 28366 tablet_bootstrap.cc:492] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a: Bootstrap starting.
I20240503 02:18:46.707774 28367 ts_tablet_manager.cc:1398] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.016s
I20240503 02:18:46.709630 28366 tablet_bootstrap.cc:654] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:46.712639 27752 catalog_manager.cc:5484] T 6096f9bd19e749889c8e6154d88c921d P 5b5ff5d736d344869048c4155e43b56a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b5ff5d736d344869048c4155e43b56a (127.0.145.1). New cstate: current_term: 1 leader_uuid: "5b5ff5d736d344869048c4155e43b56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } health_report { overall_health: HEALTHY } } }
I20240503 02:18:46.712710 27753 catalog_manager.cc:5484] T 117bdd53a4784124a5fc9704b0755144 P fe00ee12f19c43b3aef56af3b569ee00 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2). New cstate: current_term: 1 leader_uuid: "fe00ee12f19c43b3aef56af3b569ee00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } health_report { overall_health: HEALTHY } } }
I20240503 02:18:46.715481 28366 tablet_bootstrap.cc:492] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a: No bootstrap required, opened a new log
I20240503 02:18:46.715656 28366 ts_tablet_manager.cc:1367] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240503 02:18:46.717080 28366 raft_consensus.cc:348] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:46.717278 28366 raft_consensus.cc:374] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:46.717375 28366 raft_consensus.cc:729] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5ff5d736d344869048c4155e43b56a, State: Initialized, Role: FOLLOWER
I20240503 02:18:46.717674 28366 consensus_queue.cc:260] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:46.717873 28366 raft_consensus.cc:388] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:46.717948 28366 raft_consensus.cc:482] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:46.718053 28366 raft_consensus.cc:3037] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:46.720430 28366 raft_consensus.cc:504] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:46.720674 28366 leader_election.cc:302] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b5ff5d736d344869048c4155e43b56a; no voters: 
I20240503 02:18:46.720870 28366 leader_election.cc:288] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:46.720975 28370 raft_consensus.cc:2781] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:46.721166 28370 raft_consensus.cc:686] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Becoming Leader. State: Replica: 5b5ff5d736d344869048c4155e43b56a, State: Running, Role: LEADER
I20240503 02:18:46.721335 28366 ts_tablet_manager.cc:1398] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.001s
I20240503 02:18:46.722016 28370 consensus_queue.cc:237] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:46.723981 27752 catalog_manager.cc:5484] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b5ff5d736d344869048c4155e43b56a (127.0.145.1). New cstate: current_term: 1 leader_uuid: "5b5ff5d736d344869048c4155e43b56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } health_report { overall_health: HEALTHY } } }
I20240503 02:18:47.242242 27752 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54568:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:18:47.250329 28006 tablet_service.cc:1467] Processing CreateTablet for tablet 04b01a4f7ef5461a956c288f9a9458d0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2b1c093c7c854f608a38cea5c1d13449]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:18:47.250376 28005 tablet_service.cc:1467] Processing CreateTablet for tablet 7875d314f4a54e9cb2790b6a31f1ee8c (DEFAULT_TABLE table=rebalance_test_table_1 [id=2b1c093c7c854f608a38cea5c1d13449]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:18:47.250993 27874 tablet_service.cc:1467] Processing CreateTablet for tablet b1a551ede9bb43c48b062df3c404bdd1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2b1c093c7c854f608a38cea5c1d13449]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:18:47.251369 27874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a551ede9bb43c48b062df3c404bdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:47.251400 28005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7875d314f4a54e9cb2790b6a31f1ee8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:47.251631 28006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b01a4f7ef5461a956c288f9a9458d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:47.257995 28407 tablet_bootstrap.cc:492] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a: Bootstrap starting.
I20240503 02:18:47.258265 28406 tablet_bootstrap.cc:492] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00: Bootstrap starting.
I20240503 02:18:47.259553 28407 tablet_bootstrap.cc:654] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:47.259756 28406 tablet_bootstrap.cc:654] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:47.261303 28407 tablet_bootstrap.cc:492] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a: No bootstrap required, opened a new log
I20240503 02:18:47.261422 28407 ts_tablet_manager.cc:1367] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:18:47.261510 28406 tablet_bootstrap.cc:492] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00: No bootstrap required, opened a new log
I20240503 02:18:47.261636 28406 ts_tablet_manager.cc:1367] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:18:47.262068 28407 raft_consensus.cc:348] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:47.262277 28407 raft_consensus.cc:374] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:47.262377 28407 raft_consensus.cc:729] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5ff5d736d344869048c4155e43b56a, State: Initialized, Role: FOLLOWER
I20240503 02:18:47.262450 28406 raft_consensus.cc:348] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:47.262683 28406 raft_consensus.cc:374] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:47.262712 28407 consensus_queue.cc:260] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:47.262789 28406 raft_consensus.cc:729] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Initialized, Role: FOLLOWER
I20240503 02:18:47.262907 28407 raft_consensus.cc:388] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:47.262996 28407 raft_consensus.cc:482] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:47.263125 28407 raft_consensus.cc:3037] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:47.263110 28406 consensus_queue.cc:260] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:47.263284 28406 raft_consensus.cc:388] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:47.263368 28406 raft_consensus.cc:482] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:47.263471 28406 raft_consensus.cc:3037] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:47.265267 28407 raft_consensus.cc:504] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:47.265378 28406 raft_consensus.cc:504] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:47.265489 28407 leader_election.cc:302] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b5ff5d736d344869048c4155e43b56a; no voters: 
I20240503 02:18:47.265576 28406 leader_election.cc:302] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe00ee12f19c43b3aef56af3b569ee00; no voters: 
I20240503 02:18:47.265712 28407 leader_election.cc:288] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:47.265811 28370 raft_consensus.cc:2781] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:47.266255 28370 raft_consensus.cc:686] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Becoming Leader. State: Replica: 5b5ff5d736d344869048c4155e43b56a, State: Running, Role: LEADER
I20240503 02:18:47.266589 28370 consensus_queue.cc:237] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:47.268057 28407 ts_tablet_manager.cc:1398] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.002s
I20240503 02:18:47.268394 28406 leader_election.cc:288] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:47.268546 28410 raft_consensus.cc:2781] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:47.269346 27752 catalog_manager.cc:5484] T b1a551ede9bb43c48b062df3c404bdd1 P 5b5ff5d736d344869048c4155e43b56a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b5ff5d736d344869048c4155e43b56a (127.0.145.1). New cstate: current_term: 1 leader_uuid: "5b5ff5d736d344869048c4155e43b56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } health_report { overall_health: HEALTHY } } }
I20240503 02:18:47.271201 28406 ts_tablet_manager.cc:1398] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.007s
I20240503 02:18:47.271499 28406 tablet_bootstrap.cc:492] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00: Bootstrap starting.
I20240503 02:18:47.272909 28406 tablet_bootstrap.cc:654] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:47.274152 28410 raft_consensus.cc:686] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Becoming Leader. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Running, Role: LEADER
I20240503 02:18:47.274564 28410 consensus_queue.cc:237] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:47.275117 28406 tablet_bootstrap.cc:492] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00: No bootstrap required, opened a new log
I20240503 02:18:47.275296 28406 ts_tablet_manager.cc:1367] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.001s
I20240503 02:18:47.276073 28406 raft_consensus.cc:348] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:47.276271 28406 raft_consensus.cc:374] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:47.276376 28406 raft_consensus.cc:729] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Initialized, Role: FOLLOWER
I20240503 02:18:47.276630 28406 consensus_queue.cc:260] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:47.276788 28406 raft_consensus.cc:388] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:47.276888 28406 raft_consensus.cc:482] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:47.276988 28406 raft_consensus.cc:3037] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:47.277925 27753 catalog_manager.cc:5484] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2). New cstate: current_term: 1 leader_uuid: "fe00ee12f19c43b3aef56af3b569ee00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } health_report { overall_health: HEALTHY } } }
I20240503 02:18:47.279456 28406 raft_consensus.cc:504] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:47.279690 28406 leader_election.cc:302] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe00ee12f19c43b3aef56af3b569ee00; no voters: 
I20240503 02:18:47.279894 28406 leader_election.cc:288] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:47.280293 28406 ts_tablet_manager.cc:1398] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.002s
I20240503 02:18:47.280336 28410 raft_consensus.cc:2781] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:47.280511 28410 raft_consensus.cc:686] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Becoming Leader. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Running, Role: LEADER
I20240503 02:18:47.280750 28410 consensus_queue.cc:237] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:47.283008 27753 catalog_manager.cc:5484] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2). New cstate: current_term: 1 leader_uuid: "fe00ee12f19c43b3aef56af3b569ee00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } health_report { overall_health: HEALTHY } } }
I20240503 02:18:47.791363 27753 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54568:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:18:47.799690 28005 tablet_service.cc:1467] Processing CreateTablet for tablet 6a6678b8bc4d4867a39bf894d2f8d0f3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=50e4a3d399854e64b553e1074d4529b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:18:47.799786 27874 tablet_service.cc:1467] Processing CreateTablet for tablet 56b204d83cb148dc841afb1e2aaedbba (DEFAULT_TABLE table=rebalance_test_table_2 [id=50e4a3d399854e64b553e1074d4529b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:18:47.799786 27875 tablet_service.cc:1467] Processing CreateTablet for tablet 189a5b9a4d2943548d9fbcd25e6cf0f7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=50e4a3d399854e64b553e1074d4529b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:18:47.800186 28005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a6678b8bc4d4867a39bf894d2f8d0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:47.800262 27874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b204d83cb148dc841afb1e2aaedbba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:47.800487 27875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 189a5b9a4d2943548d9fbcd25e6cf0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:47.805224 28444 tablet_bootstrap.cc:492] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00: Bootstrap starting.
I20240503 02:18:47.806898 28444 tablet_bootstrap.cc:654] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:47.807098 28445 tablet_bootstrap.cc:492] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a: Bootstrap starting.
I20240503 02:18:47.808627 28445 tablet_bootstrap.cc:654] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:47.808820 28444 tablet_bootstrap.cc:492] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00: No bootstrap required, opened a new log
I20240503 02:18:47.808984 28444 ts_tablet_manager.cc:1367] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:18:47.809760 28444 raft_consensus.cc:348] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:47.809931 28444 raft_consensus.cc:374] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:47.810041 28444 raft_consensus.cc:729] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Initialized, Role: FOLLOWER
I20240503 02:18:47.810276 28444 consensus_queue.cc:260] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:47.810467 28444 raft_consensus.cc:388] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:47.810505 28445 tablet_bootstrap.cc:492] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a: No bootstrap required, opened a new log
I20240503 02:18:47.810571 28444 raft_consensus.cc:482] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:47.810659 28445 ts_tablet_manager.cc:1367] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.003s
I20240503 02:18:47.810729 28444 raft_consensus.cc:3037] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:47.811409 28445 raft_consensus.cc:348] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:47.811578 28445 raft_consensus.cc:374] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:47.811659 28445 raft_consensus.cc:729] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5ff5d736d344869048c4155e43b56a, State: Initialized, Role: FOLLOWER
I20240503 02:18:47.811923 28445 consensus_queue.cc:260] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:47.812114 28445 raft_consensus.cc:388] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:47.812191 28445 raft_consensus.cc:482] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:47.812296 28445 raft_consensus.cc:3037] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:47.813484 28444 raft_consensus.cc:504] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:47.813741 28444 leader_election.cc:302] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe00ee12f19c43b3aef56af3b569ee00; no voters: 
I20240503 02:18:47.813956 28444 leader_election.cc:288] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:47.814056 28410 raft_consensus.cc:2781] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:47.814507 28410 raft_consensus.cc:686] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Becoming Leader. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Running, Role: LEADER
I20240503 02:18:47.814818 28410 consensus_queue.cc:237] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:47.815070 28445 raft_consensus.cc:504] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:47.815393 28445 leader_election.cc:302] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b5ff5d736d344869048c4155e43b56a; no voters: 
I20240503 02:18:47.815915 28445 leader_election.cc:288] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:47.816175 28451 raft_consensus.cc:2781] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:47.821375 28444 ts_tablet_manager.cc:1398] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00: Time spent starting tablet: real 0.012s	user 0.012s	sys 0.000s
I20240503 02:18:47.822232 28445 ts_tablet_manager.cc:1398] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.004s
I20240503 02:18:47.822547 28445 tablet_bootstrap.cc:492] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a: Bootstrap starting.
I20240503 02:18:47.822474 28451 raft_consensus.cc:686] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Becoming Leader. State: Replica: 5b5ff5d736d344869048c4155e43b56a, State: Running, Role: LEADER
I20240503 02:18:47.822875 28451 consensus_queue.cc:237] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:47.824357 28445 tablet_bootstrap.cc:654] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:47.824911 27753 catalog_manager.cc:5484] T 6a6678b8bc4d4867a39bf894d2f8d0f3 P fe00ee12f19c43b3aef56af3b569ee00 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2). New cstate: current_term: 1 leader_uuid: "fe00ee12f19c43b3aef56af3b569ee00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } health_report { overall_health: HEALTHY } } }
I20240503 02:18:47.827514 28445 tablet_bootstrap.cc:492] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a: No bootstrap required, opened a new log
I20240503 02:18:47.827669 28445 ts_tablet_manager.cc:1367] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240503 02:18:47.827729 27752 catalog_manager.cc:5484] T 189a5b9a4d2943548d9fbcd25e6cf0f7 P 5b5ff5d736d344869048c4155e43b56a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b5ff5d736d344869048c4155e43b56a (127.0.145.1). New cstate: current_term: 1 leader_uuid: "5b5ff5d736d344869048c4155e43b56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } health_report { overall_health: HEALTHY } } }
I20240503 02:18:47.828338 28445 raft_consensus.cc:348] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:47.828492 28445 raft_consensus.cc:374] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:47.828570 28445 raft_consensus.cc:729] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5ff5d736d344869048c4155e43b56a, State: Initialized, Role: FOLLOWER
I20240503 02:18:47.828805 28445 consensus_queue.cc:260] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:47.828959 28445 raft_consensus.cc:388] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:47.829085 28445 raft_consensus.cc:482] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:47.829183 28445 raft_consensus.cc:3037] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:47.831683 28445 raft_consensus.cc:504] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:47.831892 28445 leader_election.cc:302] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b5ff5d736d344869048c4155e43b56a; no voters: 
I20240503 02:18:47.832118 28445 leader_election.cc:288] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:47.832361 28453 raft_consensus.cc:2781] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:47.832504 28453 raft_consensus.cc:686] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Becoming Leader. State: Replica: 5b5ff5d736d344869048c4155e43b56a, State: Running, Role: LEADER
I20240503 02:18:47.832767 28453 consensus_queue.cc:237] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:47.833897 28445 ts_tablet_manager.cc:1398] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.001s
I20240503 02:18:47.835263 27752 catalog_manager.cc:5484] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b5ff5d736d344869048c4155e43b56a (127.0.145.1). New cstate: current_term: 1 leader_uuid: "5b5ff5d736d344869048c4155e43b56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } health_report { overall_health: HEALTHY } } }
I20240503 02:18:49.238209 27752 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54568:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:18:49.246423 28005 tablet_service.cc:1467] Processing CreateTablet for tablet d8349bd8ed5d49d4a4f6cd99239b1ebc (DEFAULT_TABLE table=rebalance_test_table_3 [id=bfa3aab996bd4ee4a61113cff9ce2cb1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:18:49.246515 28006 tablet_service.cc:1467] Processing CreateTablet for tablet b9e2225705cb41dbaeccbe83e1dc03d1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bfa3aab996bd4ee4a61113cff9ce2cb1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:18:49.246991 28005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8349bd8ed5d49d4a4f6cd99239b1ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:49.247221 28006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9e2225705cb41dbaeccbe83e1dc03d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:49.247356 27875 tablet_service.cc:1467] Processing CreateTablet for tablet e65563b9a9314a6a89ee51d006513dad (DEFAULT_TABLE table=rebalance_test_table_3 [id=bfa3aab996bd4ee4a61113cff9ce2cb1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:18:49.247735 27875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65563b9a9314a6a89ee51d006513dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:49.252311 28485 tablet_bootstrap.cc:492] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00: Bootstrap starting.
I20240503 02:18:49.254285 28485 tablet_bootstrap.cc:654] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:49.255585 28486 tablet_bootstrap.cc:492] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a: Bootstrap starting.
I20240503 02:18:49.255990 28485 tablet_bootstrap.cc:492] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00: No bootstrap required, opened a new log
I20240503 02:18:49.256105 28485 ts_tablet_manager.cc:1367] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:18:49.256932 28486 tablet_bootstrap.cc:654] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:49.256870 28485 raft_consensus.cc:348] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:49.257014 28485 raft_consensus.cc:374] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:49.257103 28485 raft_consensus.cc:729] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Initialized, Role: FOLLOWER
I20240503 02:18:49.257337 28485 consensus_queue.cc:260] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:49.257527 28485 raft_consensus.cc:388] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:49.257601 28485 raft_consensus.cc:482] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:49.257719 28485 raft_consensus.cc:3037] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:49.258735 28486 tablet_bootstrap.cc:492] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a: No bootstrap required, opened a new log
I20240503 02:18:49.258864 28486 ts_tablet_manager.cc:1367] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:18:49.259441 28486 raft_consensus.cc:348] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:49.259594 28486 raft_consensus.cc:374] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:49.259694 28486 raft_consensus.cc:729] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5ff5d736d344869048c4155e43b56a, State: Initialized, Role: FOLLOWER
I20240503 02:18:49.259861 28485 raft_consensus.cc:504] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:49.259903 28486 consensus_queue.cc:260] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:49.260062 28486 raft_consensus.cc:388] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:49.260066 28485 leader_election.cc:302] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe00ee12f19c43b3aef56af3b569ee00; no voters: 
I20240503 02:18:49.260145 28486 raft_consensus.cc:482] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:49.260257 28486 raft_consensus.cc:3037] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:49.260470 28485 leader_election.cc:288] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:49.260607 28489 raft_consensus.cc:2781] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:49.261890 28489 raft_consensus.cc:686] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Becoming Leader. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Running, Role: LEADER
I20240503 02:18:49.262197 28489 consensus_queue.cc:237] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:49.263034 28486 raft_consensus.cc:504] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:49.263294 28486 leader_election.cc:302] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b5ff5d736d344869048c4155e43b56a; no voters: 
I20240503 02:18:49.264665 27752 catalog_manager.cc:5484] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2). New cstate: current_term: 1 leader_uuid: "fe00ee12f19c43b3aef56af3b569ee00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } health_report { overall_health: HEALTHY } } }
I20240503 02:18:49.266867 28493 raft_consensus.cc:2781] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:49.267153 28486 leader_election.cc:288] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:49.268049 28485 ts_tablet_manager.cc:1398] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.004s
I20240503 02:18:49.268436 28485 tablet_bootstrap.cc:492] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00: Bootstrap starting.
I20240503 02:18:49.269104 28493 raft_consensus.cc:686] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Becoming Leader. State: Replica: 5b5ff5d736d344869048c4155e43b56a, State: Running, Role: LEADER
I20240503 02:18:49.269536 28493 consensus_queue.cc:237] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } }
I20240503 02:18:49.271128 28485 tablet_bootstrap.cc:654] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:49.271152 28486 ts_tablet_manager.cc:1398] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.002s
I20240503 02:18:49.273310 27752 catalog_manager.cc:5484] T e65563b9a9314a6a89ee51d006513dad P 5b5ff5d736d344869048c4155e43b56a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b5ff5d736d344869048c4155e43b56a (127.0.145.1). New cstate: current_term: 1 leader_uuid: "5b5ff5d736d344869048c4155e43b56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } health_report { overall_health: HEALTHY } } }
I20240503 02:18:49.274228 28485 tablet_bootstrap.cc:492] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00: No bootstrap required, opened a new log
I20240503 02:18:49.274392 28485 ts_tablet_manager.cc:1367] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240503 02:18:49.275159 28485 raft_consensus.cc:348] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:49.275357 28485 raft_consensus.cc:374] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:49.275444 28485 raft_consensus.cc:729] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Initialized, Role: FOLLOWER
I20240503 02:18:49.275691 28485 consensus_queue.cc:260] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:49.275862 28485 raft_consensus.cc:388] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:49.275983 28485 raft_consensus.cc:482] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:49.276077 28485 raft_consensus.cc:3037] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:49.278168 28485 raft_consensus.cc:504] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:49.278378 28485 leader_election.cc:302] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe00ee12f19c43b3aef56af3b569ee00; no voters: 
I20240503 02:18:49.278616 28485 leader_election.cc:288] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:49.278671 28489 raft_consensus.cc:2781] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:49.278862 28489 raft_consensus.cc:686] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Becoming Leader. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Running, Role: LEADER
I20240503 02:18:49.279127 28485 ts_tablet_manager.cc:1398] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:18:49.279125 28489 consensus_queue.cc:237] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } }
I20240503 02:18:49.281786 27754 catalog_manager.cc:5484] T b9e2225705cb41dbaeccbe83e1dc03d1 P fe00ee12f19c43b3aef56af3b569ee00 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2). New cstate: current_term: 1 leader_uuid: "fe00ee12f19c43b3aef56af3b569ee00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } health_report { overall_health: HEALTHY } } }
I20240503 02:18:49.781869   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.3:36141
--local_ip_for_outbound_sockets=127.0.145.3
--tserver_master_addrs=127.0.145.62:43077
--webserver_port=34261
--webserver_interface=127.0.145.3
--builtin_ntp_servers=127.0.145.20:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:18:50.116412 28522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:18:50.116708 28522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:18:50.116811 28522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:18:50.124246 28522 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:18:50.124377 28522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:18:50.124568 28522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.3
I20240503 02:18:50.133831 28522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.3:36141
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.145.3
--webserver_port=34261
--tserver_master_addrs=127.0.145.62:43077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.3
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:18:50.135326 28522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:18:50.137094 28522 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:18:50.149818 28530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:50.150312 28528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:50.151021 28527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:18:50.151707 28522 server_base.cc:1034] running on GCE node
I20240503 02:18:50.152283 28522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:18:50.153230 28522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:18:50.154426 28522 hybrid_clock.cc:648] HybridClock initialized: now 1714702730154381 us; error 39 us; skew 500 ppm
I20240503 02:18:50.154811 28522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:18:50.157001 28522 webserver.cc:460] Webserver started at http://127.0.145.3:34261/ using document root <none> and password file <none>
I20240503 02:18:50.157730 28522 fs_manager.cc:359] Metadata directory not provided
I20240503 02:18:50.157850 28522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:18:50.163496 28522 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:18:50.166443 28536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:50.167821 28522 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240503 02:18:50.167958 28522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d85cddb1910a4ca980af31262d4474dd"
format_stamp: "Formatted at 2024-05-03 02:18:42 on dist-test-slave-gq9h"
I20240503 02:18:50.168787 28522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:18:50.193281 28522 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:18:50.194137 28522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:18:50.194418 28522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:18:50.195436 28522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:18:50.197289 28543 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 02:18:50.202677 28522 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 02:18:50.202778 28522 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
I20240503 02:18:50.202888 28522 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 02:18:50.205592 28522 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 02:18:50.205686 28522 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
I20240503 02:18:50.206019 28543 tablet_bootstrap.cc:492] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd: Bootstrap starting.
I20240503 02:18:50.223762 28543 log.cc:824] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd: Log is configured to *not* fsync() on all Append() calls
I20240503 02:18:50.228932 28543 tablet_bootstrap.cc:492] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:50.229696 28543 tablet_bootstrap.cc:492] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd: Bootstrap complete.
I20240503 02:18:50.231166 28543 ts_tablet_manager.cc:1367] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd: Time spent bootstrapping tablet: real 0.025s	user 0.017s	sys 0.007s
I20240503 02:18:50.239095 28543 raft_consensus.cc:348] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } }
I20240503 02:18:50.240288 28543 raft_consensus.cc:729] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85cddb1910a4ca980af31262d4474dd, State: Initialized, Role: FOLLOWER
I20240503 02:18:50.241154 28543 consensus_queue.cc:260] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } }
I20240503 02:18:50.241426 28543 raft_consensus.cc:388] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:50.241601 28543 raft_consensus.cc:482] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:50.241814 28543 raft_consensus.cc:3037] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:18:50.247699 28543 raft_consensus.cc:504] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } }
I20240503 02:18:50.248378 28543 leader_election.cc:302] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d85cddb1910a4ca980af31262d4474dd; no voters: 
I20240503 02:18:50.249094 28543 leader_election.cc:288] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [CANDIDATE]: Term 2 election: Requested vote from peers 
I20240503 02:18:50.249328 28627 raft_consensus.cc:2781] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:18:50.251538 28543 ts_tablet_manager.cc:1398] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd: Time spent starting tablet: real 0.020s	user 0.015s	sys 0.005s
I20240503 02:18:50.251873 28627 raft_consensus.cc:686] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [term 2 LEADER]: Becoming Leader. State: Replica: d85cddb1910a4ca980af31262d4474dd, State: Running, Role: LEADER
I20240503 02:18:50.252471 28627 consensus_queue.cc:237] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } }
I20240503 02:18:50.259171 28522 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.3:36141
I20240503 02:18:50.259215 28655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.3:36141 every 8 connection(s)
I20240503 02:18:50.260470 28522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:18:50.262554   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 28522
I20240503 02:18:50.263499   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.4:40035
--local_ip_for_outbound_sockets=127.0.145.4
--tserver_master_addrs=127.0.145.62:43077
--webserver_port=39907
--webserver_interface=127.0.145.4
--builtin_ntp_servers=127.0.145.20:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:18:50.272465 28656 heartbeater.cc:347] Connected to a master server at 127.0.145.62:43077
I20240503 02:18:50.272851 28656 heartbeater.cc:464] Registering TS with master...
I20240503 02:18:50.273953 28656 heartbeater.cc:511] Master 127.0.145.62:43077 requested a full tablet report, sending...
I20240503 02:18:50.277006 27754 ts_manager.cc:194] Re-registered known tserver with Master: d85cddb1910a4ca980af31262d4474dd (127.0.145.3:36141)
I20240503 02:18:50.277410 27754 catalog_manager.cc:5484] T a8c411abd9f14362ae6a4efbdca4ac10 P d85cddb1910a4ca980af31262d4474dd reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "d85cddb1910a4ca980af31262d4474dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } health_report { overall_health: HEALTHY } } }
I20240503 02:18:50.281970 27754 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.3:45461
I20240503 02:18:50.285274 28656 heartbeater.cc:503] Master 127.0.145.62:43077 was elected leader, sending a full tablet report...
W20240503 02:18:50.605623 28660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:18:50.605911 28660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:18:50.606060 28660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:18:50.613560 28660 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:18:50.613799 28660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:18:50.613977 28660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.4
I20240503 02:18:50.623159 28660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.4:40035
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.145.4
--webserver_port=39907
--tserver_master_addrs=127.0.145.62:43077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.4
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:18:50.624486 28660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:18:50.626297 28660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:18:50.639248 28668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:50.639535 28671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:50.640184 28669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:18:50.640867 28660 server_base.cc:1034] running on GCE node
I20240503 02:18:50.641466 28660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:18:50.642392 28660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:18:50.643605 28660 hybrid_clock.cc:648] HybridClock initialized: now 1714702730643543 us; error 56 us; skew 500 ppm
I20240503 02:18:50.643922 28660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:18:50.646320 28660 webserver.cc:460] Webserver started at http://127.0.145.4:39907/ using document root <none> and password file <none>
I20240503 02:18:50.647159 28660 fs_manager.cc:359] Metadata directory not provided
I20240503 02:18:50.647291 28660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:18:50.652966 28660 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240503 02:18:50.656415 28677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:50.657827 28660 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240503 02:18:50.657974 28660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "57d37d4403774fb8b57c63a6d53f7dff"
format_stamp: "Formatted at 2024-05-03 02:18:42 on dist-test-slave-gq9h"
I20240503 02:18:50.658852 28660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:18:50.684226 28660 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:18:50.685056 28660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:18:50.685317 28660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:18:50.686264 28660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:18:50.687783 28660 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:18:50.687857 28660 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:50.687956 28660 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:18:50.688000 28660 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:50.719120 28660 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.4:40035
I20240503 02:18:50.719158 28789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.4:40035 every 8 connection(s)
I20240503 02:18:50.720593 28660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:18:50.727563   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 28660
I20240503 02:18:50.733788 28790 heartbeater.cc:347] Connected to a master server at 127.0.145.62:43077
I20240503 02:18:50.734144 28790 heartbeater.cc:464] Registering TS with master...
I20240503 02:18:50.735213 28790 heartbeater.cc:511] Master 127.0.145.62:43077 requested a full tablet report, sending...
I20240503 02:18:50.736829 27752 ts_manager.cc:194] Re-registered known tserver with Master: 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4:40035)
I20240503 02:18:50.737701 27752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.4:39509
I20240503 02:18:51.209491 28725 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:51.219786 27875 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:51.221261 28006 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:51.232091 28586 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:51.395059 28725 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:51.402979 28006 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:51.405272 27875 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:51.407886 28586 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:51.594018 28725 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:51.600486 28586 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:51.617842 27875 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:51.619442 28006 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:51.691224 28026 consensus_queue.cc:237] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } }
I20240503 02:18:51.693997 28907 raft_consensus.cc:2932] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } } }
I20240503 02:18:51.702134 27753 catalog_manager.cc:5484] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 reported cstate change: config changed from index -1 to 2, NON_VOTER 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) added. New cstate: current_term: 1 leader_uuid: "fe00ee12f19c43b3aef56af3b569ee00" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:18:51.704788 27962 consensus_peers.cc:485] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 -> Peer 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4:40035): Couldn't send request to peer 57d37d4403774fb8b57c63a6d53f7dff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7875d314f4a54e9cb2790b6a31f1ee8c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:51.712203 27895 consensus_queue.cc:237] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: true } }
I20240503 02:18:51.714883 28920 raft_consensus.cc:2932] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER d85cddb1910a4ca980af31262d4474dd (127.0.145.3) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: true } } }
I20240503 02:18:51.721400 27753 catalog_manager.cc:5484] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a reported cstate change: config changed from index -1 to 3, NON_VOTER d85cddb1910a4ca980af31262d4474dd (127.0.145.3) added. New cstate: current_term: 1 leader_uuid: "5b5ff5d736d344869048c4155e43b56a" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:18:51.724171 27828 consensus_peers.cc:485] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a -> Peer d85cddb1910a4ca980af31262d4474dd (127.0.145.3:36141): Couldn't send request to peer d85cddb1910a4ca980af31262d4474dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56b204d83cb148dc841afb1e2aaedbba. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:51.740381 28790 heartbeater.cc:503] Master 127.0.145.62:43077 was elected leader, sending a full tablet report...
I20240503 02:18:51.742867 28026 consensus_queue.cc:237] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } }
I20240503 02:18:51.744437 28906 raft_consensus.cc:2932] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } } }
W20240503 02:18:51.744841 27962 consensus_peers.cc:485] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 -> Peer 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4:40035): Couldn't send request to peer 57d37d4403774fb8b57c63a6d53f7dff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8349bd8ed5d49d4a4f6cd99239b1ebc. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:51.759459 27753 catalog_manager.cc:5484] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 reported cstate change: config changed from index -1 to 3, NON_VOTER 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) added. New cstate: current_term: 1 leader_uuid: "fe00ee12f19c43b3aef56af3b569ee00" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:51.761683 27895 consensus_queue.cc:237] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } }
I20240503 02:18:51.763648 28919 raft_consensus.cc:2932] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } } }
I20240503 02:18:51.769807 27753 catalog_manager.cc:5484] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a reported cstate change: config changed from index -1 to 3, NON_VOTER 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) added. New cstate: current_term: 1 leader_uuid: "5b5ff5d736d344869048c4155e43b56a" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:18:51.770164 27831 consensus_peers.cc:485] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a -> Peer 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4:40035): Couldn't send request to peer 57d37d4403774fb8b57c63a6d53f7dff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e833cff4caf947a986d4abb1df7ef4fb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:51.779621 28026 consensus_queue.cc:237] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: true } }
I20240503 02:18:51.781203 28928 raft_consensus.cc:2932] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER d85cddb1910a4ca980af31262d4474dd (127.0.145.3) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: true } } }
I20240503 02:18:51.786265 27753 catalog_manager.cc:5484] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 reported cstate change: config changed from index -1 to 3, NON_VOTER d85cddb1910a4ca980af31262d4474dd (127.0.145.3) added. New cstate: current_term: 1 leader_uuid: "fe00ee12f19c43b3aef56af3b569ee00" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:18:51.788391 27959 consensus_peers.cc:485] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 -> Peer d85cddb1910a4ca980af31262d4474dd (127.0.145.3:36141): Couldn't send request to peer d85cddb1910a4ca980af31262d4474dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04b01a4f7ef5461a956c288f9a9458d0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:52.080279 28991 ts_tablet_manager.cc:902] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff: Initiating tablet copy from peer fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2:35911)
I20240503 02:18:52.082590 28991 tablet_copy_client.cc:308] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35911
I20240503 02:18:52.089324 28046 tablet_copy_service.cc:140] P fe00ee12f19c43b3aef56af3b569ee00: Received BeginTabletCopySession request for tablet 7875d314f4a54e9cb2790b6a31f1ee8c from peer 57d37d4403774fb8b57c63a6d53f7dff ({username='slave'} at 127.0.145.4:55969)
I20240503 02:18:52.089586 28046 tablet_copy_service.cc:161] P fe00ee12f19c43b3aef56af3b569ee00: Beginning new tablet copy session on tablet 7875d314f4a54e9cb2790b6a31f1ee8c from peer 57d37d4403774fb8b57c63a6d53f7dff at {username='slave'} at 127.0.145.4:55969: session id = 57d37d4403774fb8b57c63a6d53f7dff-7875d314f4a54e9cb2790b6a31f1ee8c
I20240503 02:18:52.092592 28046 tablet_copy_source_session.cc:205] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:18:52.097394 28991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7875d314f4a54e9cb2790b6a31f1ee8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:52.100423 28997 ts_tablet_manager.cc:902] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd: Initiating tablet copy from peer 5b5ff5d736d344869048c4155e43b56a (127.0.145.1:38711)
I20240503 02:18:52.103622 28997 tablet_copy_client.cc:308] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:38711
I20240503 02:18:52.108984 28991 tablet_copy_client.cc:783] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff: tablet copy: Starting download of 0 data blocks...
I20240503 02:18:52.109387 28991 tablet_copy_client.cc:647] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:52.110565 27915 tablet_copy_service.cc:140] P 5b5ff5d736d344869048c4155e43b56a: Received BeginTabletCopySession request for tablet 56b204d83cb148dc841afb1e2aaedbba from peer d85cddb1910a4ca980af31262d4474dd ({username='slave'} at 127.0.145.3:46003)
I20240503 02:18:52.110841 27915 tablet_copy_service.cc:161] P 5b5ff5d736d344869048c4155e43b56a: Beginning new tablet copy session on tablet 56b204d83cb148dc841afb1e2aaedbba from peer d85cddb1910a4ca980af31262d4474dd at {username='slave'} at 127.0.145.3:46003: session id = d85cddb1910a4ca980af31262d4474dd-56b204d83cb148dc841afb1e2aaedbba
I20240503 02:18:52.111802 28991 tablet_copy_client.cc:523] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:18:52.113528 27915 tablet_copy_source_session.cc:205] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:18:52.115061 28991 tablet_bootstrap.cc:492] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff: Bootstrap starting.
I20240503 02:18:52.116101 28997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b204d83cb148dc841afb1e2aaedbba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:52.121555 28991 log.cc:824] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff: Log is configured to *not* fsync() on all Append() calls
I20240503 02:18:52.121788 28997 tablet_copy_client.cc:783] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd: tablet copy: Starting download of 0 data blocks...
I20240503 02:18:52.122205 28997 tablet_copy_client.cc:647] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:52.124680 28997 tablet_copy_client.cc:523] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:18:52.125242 28991 tablet_bootstrap.cc:492] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:52.125674 28991 tablet_bootstrap.cc:492] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff: Bootstrap complete.
I20240503 02:18:52.126060 28991 ts_tablet_manager.cc:1367] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff: Time spent bootstrapping tablet: real 0.011s	user 0.007s	sys 0.000s
I20240503 02:18:52.127455 28997 tablet_bootstrap.cc:492] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd: Bootstrap starting.
I20240503 02:18:52.132138 28991 raft_consensus.cc:348] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } }
I20240503 02:18:52.132550 28991 raft_consensus.cc:729] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57d37d4403774fb8b57c63a6d53f7dff, State: Initialized, Role: LEARNER
I20240503 02:18:52.133013 28991 consensus_queue.cc:260] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } }
I20240503 02:18:52.134752 28991 ts_tablet_manager.cc:1398] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.005s
I20240503 02:18:52.135962 28046 tablet_copy_service.cc:342] P fe00ee12f19c43b3aef56af3b569ee00: Request end of tablet copy session 57d37d4403774fb8b57c63a6d53f7dff-7875d314f4a54e9cb2790b6a31f1ee8c received from {username='slave'} at 127.0.145.4:55969
I20240503 02:18:52.136157 28046 tablet_copy_service.cc:433] P fe00ee12f19c43b3aef56af3b569ee00: ending tablet copy session 57d37d4403774fb8b57c63a6d53f7dff-7875d314f4a54e9cb2790b6a31f1ee8c on tablet 7875d314f4a54e9cb2790b6a31f1ee8c with peer 57d37d4403774fb8b57c63a6d53f7dff
I20240503 02:18:52.139283 28997 tablet_bootstrap.cc:492] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:52.139714 28997 tablet_bootstrap.cc:492] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd: Bootstrap complete.
I20240503 02:18:52.139946 28997 ts_tablet_manager.cc:1367] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.006s
I20240503 02:18:52.140749 28997 raft_consensus.cc:348] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: true } }
I20240503 02:18:52.140919 28997 raft_consensus.cc:729] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d85cddb1910a4ca980af31262d4474dd, State: Initialized, Role: LEARNER
I20240503 02:18:52.141140 28997 consensus_queue.cc:260] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: true } }
I20240503 02:18:52.141851 28997 ts_tablet_manager.cc:1398] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:18:52.142771 27915 tablet_copy_service.cc:342] P 5b5ff5d736d344869048c4155e43b56a: Request end of tablet copy session d85cddb1910a4ca980af31262d4474dd-56b204d83cb148dc841afb1e2aaedbba received from {username='slave'} at 127.0.145.3:46003
I20240503 02:18:52.142928 27915 tablet_copy_service.cc:433] P 5b5ff5d736d344869048c4155e43b56a: ending tablet copy session d85cddb1910a4ca980af31262d4474dd-56b204d83cb148dc841afb1e2aaedbba on tablet 56b204d83cb148dc841afb1e2aaedbba with peer d85cddb1910a4ca980af31262d4474dd
I20240503 02:18:52.275194 28991 ts_tablet_manager.cc:902] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff: Initiating tablet copy from peer 5b5ff5d736d344869048c4155e43b56a (127.0.145.1:38711)
I20240503 02:18:52.276378 28991 tablet_copy_client.cc:308] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:38711
I20240503 02:18:52.281409 27915 tablet_copy_service.cc:140] P 5b5ff5d736d344869048c4155e43b56a: Received BeginTabletCopySession request for tablet e833cff4caf947a986d4abb1df7ef4fb from peer 57d37d4403774fb8b57c63a6d53f7dff ({username='slave'} at 127.0.145.4:42649)
I20240503 02:18:52.281579 27915 tablet_copy_service.cc:161] P 5b5ff5d736d344869048c4155e43b56a: Beginning new tablet copy session on tablet e833cff4caf947a986d4abb1df7ef4fb from peer 57d37d4403774fb8b57c63a6d53f7dff at {username='slave'} at 127.0.145.4:42649: session id = 57d37d4403774fb8b57c63a6d53f7dff-e833cff4caf947a986d4abb1df7ef4fb
I20240503 02:18:52.283516 27915 tablet_copy_source_session.cc:205] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:18:52.284637 28991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e833cff4caf947a986d4abb1df7ef4fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:52.290750 28991 tablet_copy_client.cc:783] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff: tablet copy: Starting download of 0 data blocks...
I20240503 02:18:52.291009 28991 tablet_copy_client.cc:647] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:52.292495 28991 tablet_copy_client.cc:523] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:18:52.295212 28991 tablet_bootstrap.cc:492] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff: Bootstrap starting.
I20240503 02:18:52.305655 28991 tablet_bootstrap.cc:492] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:52.306074 28991 tablet_bootstrap.cc:492] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff: Bootstrap complete.
I20240503 02:18:52.306322 28991 ts_tablet_manager.cc:1367] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.006s
I20240503 02:18:52.307073 28991 raft_consensus.cc:348] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } }
I20240503 02:18:52.307245 28991 raft_consensus.cc:729] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57d37d4403774fb8b57c63a6d53f7dff, State: Initialized, Role: LEARNER
I20240503 02:18:52.307426 28991 consensus_queue.cc:260] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } }
I20240503 02:18:52.307864 28991 ts_tablet_manager.cc:1398] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240503 02:18:52.308758 27915 tablet_copy_service.cc:342] P 5b5ff5d736d344869048c4155e43b56a: Request end of tablet copy session 57d37d4403774fb8b57c63a6d53f7dff-e833cff4caf947a986d4abb1df7ef4fb received from {username='slave'} at 127.0.145.4:42649
I20240503 02:18:52.308866 27915 tablet_copy_service.cc:433] P 5b5ff5d736d344869048c4155e43b56a: ending tablet copy session 57d37d4403774fb8b57c63a6d53f7dff-e833cff4caf947a986d4abb1df7ef4fb on tablet e833cff4caf947a986d4abb1df7ef4fb with peer 57d37d4403774fb8b57c63a6d53f7dff
I20240503 02:18:52.342000 28997 ts_tablet_manager.cc:902] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd: Initiating tablet copy from peer fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2:35911)
I20240503 02:18:52.349139 28997 tablet_copy_client.cc:308] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35911
I20240503 02:18:52.354074 28046 tablet_copy_service.cc:140] P fe00ee12f19c43b3aef56af3b569ee00: Received BeginTabletCopySession request for tablet 04b01a4f7ef5461a956c288f9a9458d0 from peer d85cddb1910a4ca980af31262d4474dd ({username='slave'} at 127.0.145.3:55685)
I20240503 02:18:52.354215 28046 tablet_copy_service.cc:161] P fe00ee12f19c43b3aef56af3b569ee00: Beginning new tablet copy session on tablet 04b01a4f7ef5461a956c288f9a9458d0 from peer d85cddb1910a4ca980af31262d4474dd at {username='slave'} at 127.0.145.3:55685: session id = d85cddb1910a4ca980af31262d4474dd-04b01a4f7ef5461a956c288f9a9458d0
I20240503 02:18:52.355685 28991 ts_tablet_manager.cc:902] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff: Initiating tablet copy from peer fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2:35911)
I20240503 02:18:52.356549 28046 tablet_copy_source_session.cc:205] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:18:52.356931 28991 tablet_copy_client.cc:308] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:35911
I20240503 02:18:52.357553 28046 tablet_copy_service.cc:140] P fe00ee12f19c43b3aef56af3b569ee00: Received BeginTabletCopySession request for tablet d8349bd8ed5d49d4a4f6cd99239b1ebc from peer 57d37d4403774fb8b57c63a6d53f7dff ({username='slave'} at 127.0.145.4:55969)
I20240503 02:18:52.357676 28046 tablet_copy_service.cc:161] P fe00ee12f19c43b3aef56af3b569ee00: Beginning new tablet copy session on tablet d8349bd8ed5d49d4a4f6cd99239b1ebc from peer 57d37d4403774fb8b57c63a6d53f7dff at {username='slave'} at 127.0.145.4:55969: session id = 57d37d4403774fb8b57c63a6d53f7dff-d8349bd8ed5d49d4a4f6cd99239b1ebc
I20240503 02:18:52.357754 28997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b01a4f7ef5461a956c288f9a9458d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:52.359589 28046 tablet_copy_source_session.cc:205] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:18:52.360611 28991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8349bd8ed5d49d4a4f6cd99239b1ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:52.362345 28997 tablet_copy_client.cc:783] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd: tablet copy: Starting download of 0 data blocks...
I20240503 02:18:52.362576 28997 tablet_copy_client.cc:647] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:52.364027 28997 tablet_copy_client.cc:523] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:18:52.365293 28991 tablet_copy_client.cc:783] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff: tablet copy: Starting download of 0 data blocks...
I20240503 02:18:52.365518 28991 tablet_copy_client.cc:647] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:52.366430 28997 tablet_bootstrap.cc:492] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd: Bootstrap starting.
I20240503 02:18:52.367086 28991 tablet_copy_client.cc:523] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:18:52.369470 28991 tablet_bootstrap.cc:492] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff: Bootstrap starting.
I20240503 02:18:52.372094 28997 tablet_bootstrap.cc:492] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:52.372433 28997 tablet_bootstrap.cc:492] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd: Bootstrap complete.
I20240503 02:18:52.372678 28997 ts_tablet_manager.cc:1367] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd: Time spent bootstrapping tablet: real 0.006s	user 0.006s	sys 0.000s
I20240503 02:18:52.373363 28997 raft_consensus.cc:348] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: true } }
I20240503 02:18:52.373520 28997 raft_consensus.cc:729] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d85cddb1910a4ca980af31262d4474dd, State: Initialized, Role: LEARNER
I20240503 02:18:52.373728 28997 consensus_queue.cc:260] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: true } }
I20240503 02:18:52.374223 28997 ts_tablet_manager.cc:1398] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:18:52.375073 28046 tablet_copy_service.cc:342] P fe00ee12f19c43b3aef56af3b569ee00: Request end of tablet copy session d85cddb1910a4ca980af31262d4474dd-04b01a4f7ef5461a956c288f9a9458d0 received from {username='slave'} at 127.0.145.3:55685
I20240503 02:18:52.375218 28046 tablet_copy_service.cc:433] P fe00ee12f19c43b3aef56af3b569ee00: ending tablet copy session d85cddb1910a4ca980af31262d4474dd-04b01a4f7ef5461a956c288f9a9458d0 on tablet 04b01a4f7ef5461a956c288f9a9458d0 with peer d85cddb1910a4ca980af31262d4474dd
I20240503 02:18:52.375726 28991 tablet_bootstrap.cc:492] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:18:52.376379 28991 tablet_bootstrap.cc:492] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff: Bootstrap complete.
I20240503 02:18:52.376627 28991 ts_tablet_manager.cc:1367] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff: Time spent bootstrapping tablet: real 0.007s	user 0.008s	sys 0.001s
I20240503 02:18:52.377246 28991 raft_consensus.cc:348] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } }
I20240503 02:18:52.377435 28991 raft_consensus.cc:729] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 57d37d4403774fb8b57c63a6d53f7dff, State: Initialized, Role: LEARNER
I20240503 02:18:52.377638 28991 consensus_queue.cc:260] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: true } }
I20240503 02:18:52.378113 28991 ts_tablet_manager.cc:1398] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:18:52.378890 28046 tablet_copy_service.cc:342] P fe00ee12f19c43b3aef56af3b569ee00: Request end of tablet copy session 57d37d4403774fb8b57c63a6d53f7dff-d8349bd8ed5d49d4a4f6cd99239b1ebc received from {username='slave'} at 127.0.145.4:55969
I20240503 02:18:52.379010 28046 tablet_copy_service.cc:433] P fe00ee12f19c43b3aef56af3b569ee00: ending tablet copy session 57d37d4403774fb8b57c63a6d53f7dff-d8349bd8ed5d49d4a4f6cd99239b1ebc on tablet d8349bd8ed5d49d4a4f6cd99239b1ebc with peer 57d37d4403774fb8b57c63a6d53f7dff
I20240503 02:18:52.499158 28745 raft_consensus.cc:1203] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240503 02:18:52.566530 28606 raft_consensus.cc:1203] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
I20240503 02:18:52.676589 28745 raft_consensus.cc:1203] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
I20240503 02:18:52.828728 28745 raft_consensus.cc:1203] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
I20240503 02:18:52.934453 28606 raft_consensus.cc:1203] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
I20240503 02:18:53.077059 29048 raft_consensus.cc:1050] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00: attempting to promote NON_VOTER 57d37d4403774fb8b57c63a6d53f7dff to VOTER
I20240503 02:18:53.077980 29048 consensus_queue.cc:237] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:53.080087 28745 raft_consensus.cc:1261] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [term 1 LEARNER]: Refusing update from remote peer fe00ee12f19c43b3aef56af3b569ee00: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:18:53.080751 29047 consensus_queue.cc:1035] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:18:53.084873 29048 raft_consensus.cc:2932] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } } }
I20240503 02:18:53.086953 29060 raft_consensus.cc:1050] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a: attempting to promote NON_VOTER d85cddb1910a4ca980af31262d4474dd to VOTER
I20240503 02:18:53.087736 29060 consensus_queue.cc:237] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } }
I20240503 02:18:53.087888 28745 raft_consensus.cc:2932] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } } }
I20240503 02:18:53.089563 28606 raft_consensus.cc:1261] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [term 1 LEARNER]: Refusing update from remote peer 5b5ff5d736d344869048c4155e43b56a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:18:53.090219 28919 consensus_queue.cc:1035] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:18:53.092631 27750 catalog_manager.cc:5484] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 reported cstate change: config changed from index 2 to 3, 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe00ee12f19c43b3aef56af3b569ee00" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:53.102798 29060 raft_consensus.cc:2932] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, d85cddb1910a4ca980af31262d4474dd (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } } }
I20240503 02:18:53.105083 28919 raft_consensus.cc:1050] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a: attempting to promote NON_VOTER 57d37d4403774fb8b57c63a6d53f7dff to VOTER
I20240503 02:18:53.105682 28919 consensus_queue.cc:237] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:53.106987 28606 raft_consensus.cc:2932] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, d85cddb1910a4ca980af31262d4474dd (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } } }
I20240503 02:18:53.109745 27752 catalog_manager.cc:5484] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a reported cstate change: config changed from index 3 to 4, d85cddb1910a4ca980af31262d4474dd (127.0.145.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b5ff5d736d344869048c4155e43b56a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:53.116739 28745 raft_consensus.cc:1261] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [term 1 LEARNER]: Refusing update from remote peer 5b5ff5d736d344869048c4155e43b56a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:18:53.117350 28919 consensus_queue.cc:1035] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [LEADER]: Connected to new peer: Peer: permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:18:53.120169 29060 raft_consensus.cc:2932] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } } }
I20240503 02:18:53.120826 28745 raft_consensus.cc:2932] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } } }
I20240503 02:18:53.125718 27750 catalog_manager.cc:5484] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a reported cstate change: config changed from index 3 to 4, 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b5ff5d736d344869048c4155e43b56a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:53.134927 27895 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "56b204d83cb148dc841afb1e2aaedbba"
dest_uuid: "5b5ff5d736d344869048c4155e43b56a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43098
I20240503 02:18:53.135185 27895 raft_consensus.cc:595] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Received request to transfer leadership
I20240503 02:18:53.264783 29047 raft_consensus.cc:1050] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00: attempting to promote NON_VOTER 57d37d4403774fb8b57c63a6d53f7dff to VOTER
I20240503 02:18:53.265375 29047 consensus_queue.cc:237] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:53.267220 28745 raft_consensus.cc:1261] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [term 1 LEARNER]: Refusing update from remote peer fe00ee12f19c43b3aef56af3b569ee00: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:18:53.267674 29048 consensus_queue.cc:1035] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:18:53.270217 29047 raft_consensus.cc:2932] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } } }
I20240503 02:18:53.271241 28745 raft_consensus.cc:2932] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } } }
I20240503 02:18:53.274449 27752 catalog_manager.cc:5484] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 reported cstate change: config changed from index 3 to 4, 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe00ee12f19c43b3aef56af3b569ee00" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:53.364605 28026 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8349bd8ed5d49d4a4f6cd99239b1ebc"
dest_uuid: "fe00ee12f19c43b3aef56af3b569ee00"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40404
I20240503 02:18:53.364843 28026 raft_consensus.cc:595] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:18:53.382452 27895 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e833cff4caf947a986d4abb1df7ef4fb"
dest_uuid: "5b5ff5d736d344869048c4155e43b56a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43124
I20240503 02:18:53.382684 27895 raft_consensus.cc:595] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Received request to transfer leadership
I20240503 02:18:53.401274 28026 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7875d314f4a54e9cb2790b6a31f1ee8c"
dest_uuid: "fe00ee12f19c43b3aef56af3b569ee00"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40420
I20240503 02:18:53.401499 28026 raft_consensus.cc:595] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:18:53.419468 27895 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "56b204d83cb148dc841afb1e2aaedbba"
dest_uuid: "5b5ff5d736d344869048c4155e43b56a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43150
I20240503 02:18:53.419710 27895 raft_consensus.cc:595] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Received request to transfer leadership
I20240503 02:18:53.450460 29048 raft_consensus.cc:1050] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00: attempting to promote NON_VOTER d85cddb1910a4ca980af31262d4474dd to VOTER
I20240503 02:18:53.451090 29048 consensus_queue.cc:237] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } }
I20240503 02:18:53.452756 28606 raft_consensus.cc:1261] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [term 1 LEARNER]: Refusing update from remote peer fe00ee12f19c43b3aef56af3b569ee00: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:18:53.453258 29048 consensus_queue.cc:1035] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:18:53.455639 29047 raft_consensus.cc:2932] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, d85cddb1910a4ca980af31262d4474dd (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } } }
I20240503 02:18:53.456249 28606 raft_consensus.cc:2932] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, d85cddb1910a4ca980af31262d4474dd (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } } }
I20240503 02:18:53.459837 27750 catalog_manager.cc:5484] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 reported cstate change: config changed from index 3 to 4, d85cddb1910a4ca980af31262d4474dd (127.0.145.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe00ee12f19c43b3aef56af3b569ee00" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:53.476190 28928 raft_consensus.cc:979] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00: : Instructing follower 57d37d4403774fb8b57c63a6d53f7dff to start an election
I20240503 02:18:53.476384 28928 raft_consensus.cc:1067] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Signalling peer 57d37d4403774fb8b57c63a6d53f7dff to start an election
I20240503 02:18:53.477172 28745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7875d314f4a54e9cb2790b6a31f1ee8c"
dest_uuid: "57d37d4403774fb8b57c63a6d53f7dff"
 from {username='slave'} at 127.0.145.2:44347
I20240503 02:18:53.477452 28745 raft_consensus.cc:482] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:18:53.477614 28745 raft_consensus.cc:3037] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:18:53.479642 28745 raft_consensus.cc:504] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:53.480873 28745 leader_election.cc:288] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [CANDIDATE]: Term 2 election: Requested vote from peers fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2:35911)
I20240503 02:18:53.481458 28026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7875d314f4a54e9cb2790b6a31f1ee8c" candidate_uuid: "57d37d4403774fb8b57c63a6d53f7dff" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fe00ee12f19c43b3aef56af3b569ee00"
I20240503 02:18:53.481746 28026 raft_consensus.cc:3032] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:18:53.481837 28026 raft_consensus.cc:729] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Running, Role: LEADER
I20240503 02:18:53.482132 28026 consensus_queue.cc:260] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:53.482439 28026 raft_consensus.cc:3037] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:18:53.484319 28026 raft_consensus.cc:2454] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57d37d4403774fb8b57c63a6d53f7dff in term 2.
I20240503 02:18:53.484880 28680 leader_election.cc:302] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 57d37d4403774fb8b57c63a6d53f7dff, fe00ee12f19c43b3aef56af3b569ee00; no voters: 
I20240503 02:18:53.485272 29005 raft_consensus.cc:2781] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:18:53.485683 29005 raft_consensus.cc:686] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [term 2 LEADER]: Becoming Leader. State: Replica: 57d37d4403774fb8b57c63a6d53f7dff, State: Running, Role: LEADER
I20240503 02:18:53.486099 29005 consensus_queue.cc:237] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:53.489391 27752 catalog_manager.cc:5484] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff reported cstate change: term changed from 1 to 2, leader changed from fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2) to 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4). New cstate: current_term: 2 leader_uuid: "57d37d4403774fb8b57c63a6d53f7dff" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 02:18:53.499123 27741 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 7875d314f4a54e9cb2790b6a31f1ee8c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4:40035) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 02:18:53.599125 29060 raft_consensus.cc:979] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a: : Instructing follower d85cddb1910a4ca980af31262d4474dd to start an election
I20240503 02:18:53.599354 29060 raft_consensus.cc:1067] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Signalling peer d85cddb1910a4ca980af31262d4474dd to start an election
I20240503 02:18:53.600363 28606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "56b204d83cb148dc841afb1e2aaedbba"
dest_uuid: "d85cddb1910a4ca980af31262d4474dd"
 from {username='slave'} at 127.0.145.1:37273
I20240503 02:18:53.600637 28606 raft_consensus.cc:482] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:18:53.600770 28606 raft_consensus.cc:3037] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:18:53.602636 28606 raft_consensus.cc:504] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } }
I20240503 02:18:53.603673 28606 leader_election.cc:288] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [CANDIDATE]: Term 2 election: Requested vote from peers 5b5ff5d736d344869048c4155e43b56a (127.0.145.1:38711)
I20240503 02:18:53.604351 27895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b204d83cb148dc841afb1e2aaedbba" candidate_uuid: "d85cddb1910a4ca980af31262d4474dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5b5ff5d736d344869048c4155e43b56a"
I20240503 02:18:53.604595 27895 raft_consensus.cc:3032] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:18:53.604693 27895 raft_consensus.cc:729] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5b5ff5d736d344869048c4155e43b56a, State: Running, Role: LEADER
I20240503 02:18:53.604967 27895 consensus_queue.cc:260] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } }
I20240503 02:18:53.605262 27895 raft_consensus.cc:3037] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:18:53.607255 27895 raft_consensus.cc:2454] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d85cddb1910a4ca980af31262d4474dd in term 2.
I20240503 02:18:53.607818 28539 leader_election.cc:302] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5b5ff5d736d344869048c4155e43b56a, d85cddb1910a4ca980af31262d4474dd; no voters: 
I20240503 02:18:53.608158 29006 raft_consensus.cc:2781] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:18:53.608600 29006 raft_consensus.cc:686] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [term 2 LEADER]: Becoming Leader. State: Replica: d85cddb1910a4ca980af31262d4474dd, State: Running, Role: LEADER
I20240503 02:18:53.608927 29006 consensus_queue.cc:237] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } }
I20240503 02:18:53.612418 27752 catalog_manager.cc:5484] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd reported cstate change: term changed from 1 to 2, leader changed from 5b5ff5d736d344869048c4155e43b56a (127.0.145.1) to d85cddb1910a4ca980af31262d4474dd (127.0.145.3). New cstate: current_term: 2 leader_uuid: "d85cddb1910a4ca980af31262d4474dd" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:53.640172 28026 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04b01a4f7ef5461a956c288f9a9458d0"
dest_uuid: "fe00ee12f19c43b3aef56af3b569ee00"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40434
I20240503 02:18:53.640399 28026 raft_consensus.cc:595] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:18:53.646662 28919 raft_consensus.cc:979] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a: : Instructing follower 57d37d4403774fb8b57c63a6d53f7dff to start an election
I20240503 02:18:53.646869 28919 raft_consensus.cc:1067] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Signalling peer 57d37d4403774fb8b57c63a6d53f7dff to start an election
I20240503 02:18:53.647542 28745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e833cff4caf947a986d4abb1df7ef4fb"
dest_uuid: "57d37d4403774fb8b57c63a6d53f7dff"
 from {username='slave'} at 127.0.145.1:60131
I20240503 02:18:53.647730 28745 raft_consensus.cc:482] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:18:53.647851 28745 raft_consensus.cc:3037] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:18:53.650472 28745 raft_consensus.cc:504] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:53.651262 28745 leader_election.cc:288] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [CANDIDATE]: Term 2 election: Requested vote from peers 5b5ff5d736d344869048c4155e43b56a (127.0.145.1:38711)
I20240503 02:18:53.651831 27895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e833cff4caf947a986d4abb1df7ef4fb" candidate_uuid: "57d37d4403774fb8b57c63a6d53f7dff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5b5ff5d736d344869048c4155e43b56a"
I20240503 02:18:53.652041 27895 raft_consensus.cc:3032] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:18:53.652163 27895 raft_consensus.cc:729] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5b5ff5d736d344869048c4155e43b56a, State: Running, Role: LEADER
I20240503 02:18:53.652432 27895 consensus_queue.cc:260] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:53.652705 27895 raft_consensus.cc:3037] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:18:53.654542 27895 raft_consensus.cc:2454] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57d37d4403774fb8b57c63a6d53f7dff in term 2.
I20240503 02:18:53.654992 28680 leader_election.cc:302] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 57d37d4403774fb8b57c63a6d53f7dff, 5b5ff5d736d344869048c4155e43b56a; no voters: 
I20240503 02:18:53.655256 29005 raft_consensus.cc:2781] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:18:53.655417 29005 raft_consensus.cc:686] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [term 2 LEADER]: Becoming Leader. State: Replica: 57d37d4403774fb8b57c63a6d53f7dff, State: Running, Role: LEADER
I20240503 02:18:53.655663 29005 consensus_queue.cc:237] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:53.657969 27752 catalog_manager.cc:5484] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff reported cstate change: term changed from 1 to 2, leader changed from 5b5ff5d736d344869048c4155e43b56a (127.0.145.1) to 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4). New cstate: current_term: 2 leader_uuid: "57d37d4403774fb8b57c63a6d53f7dff" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:53.661113 28026 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8349bd8ed5d49d4a4f6cd99239b1ebc"
dest_uuid: "fe00ee12f19c43b3aef56af3b569ee00"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40454
I20240503 02:18:53.661355 28026 raft_consensus.cc:595] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:18:53.722385 29047 raft_consensus.cc:979] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00: : Instructing follower 57d37d4403774fb8b57c63a6d53f7dff to start an election
I20240503 02:18:53.722581 29047 raft_consensus.cc:1067] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Signalling peer 57d37d4403774fb8b57c63a6d53f7dff to start an election
I20240503 02:18:53.723197 28745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8349bd8ed5d49d4a4f6cd99239b1ebc"
dest_uuid: "57d37d4403774fb8b57c63a6d53f7dff"
 from {username='slave'} at 127.0.145.2:44347
I20240503 02:18:53.723400 28745 raft_consensus.cc:482] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:18:53.723502 28745 raft_consensus.cc:3037] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:18:53.725489 28745 raft_consensus.cc:504] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:53.725952 28745 leader_election.cc:288] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [CANDIDATE]: Term 2 election: Requested vote from peers fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2:35911)
I20240503 02:18:53.726513 28026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8349bd8ed5d49d4a4f6cd99239b1ebc" candidate_uuid: "57d37d4403774fb8b57c63a6d53f7dff" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fe00ee12f19c43b3aef56af3b569ee00"
I20240503 02:18:53.726769 28026 raft_consensus.cc:3032] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:18:53.726852 28026 raft_consensus.cc:729] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Running, Role: LEADER
I20240503 02:18:53.727123 28026 consensus_queue.cc:260] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:53.727367 28026 raft_consensus.cc:3037] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:18:53.729091 28026 raft_consensus.cc:2454] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57d37d4403774fb8b57c63a6d53f7dff in term 2.
I20240503 02:18:53.729496 28680 leader_election.cc:302] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 57d37d4403774fb8b57c63a6d53f7dff, fe00ee12f19c43b3aef56af3b569ee00; no voters: 
I20240503 02:18:53.729856 29190 raft_consensus.cc:2781] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:18:53.729991 29190 raft_consensus.cc:686] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [term 2 LEADER]: Becoming Leader. State: Replica: 57d37d4403774fb8b57c63a6d53f7dff, State: Running, Role: LEADER
I20240503 02:18:53.730278 29190 consensus_queue.cc:237] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:53.732555 27752 catalog_manager.cc:5484] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff reported cstate change: term changed from 1 to 2, leader changed from fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2) to 57d37d4403774fb8b57c63a6d53f7dff (127.0.145.4). New cstate: current_term: 2 leader_uuid: "57d37d4403774fb8b57c63a6d53f7dff" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:53.919957 29047 raft_consensus.cc:979] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00: : Instructing follower d85cddb1910a4ca980af31262d4474dd to start an election
I20240503 02:18:53.920193 29048 raft_consensus.cc:1067] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Signalling peer d85cddb1910a4ca980af31262d4474dd to start an election
I20240503 02:18:53.920858 28606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04b01a4f7ef5461a956c288f9a9458d0"
dest_uuid: "d85cddb1910a4ca980af31262d4474dd"
 from {username='slave'} at 127.0.145.2:43929
I20240503 02:18:53.921118 28606 raft_consensus.cc:482] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:18:53.921239 28606 raft_consensus.cc:3037] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:18:53.923949 28026 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04b01a4f7ef5461a956c288f9a9458d0"
dest_uuid: "fe00ee12f19c43b3aef56af3b569ee00"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40466
I20240503 02:18:53.924202 28026 raft_consensus.cc:595] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:18:53.924237 28606 raft_consensus.cc:504] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } }
I20240503 02:18:53.925038 28606 leader_election.cc:288] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [CANDIDATE]: Term 2 election: Requested vote from peers fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2:35911)
I20240503 02:18:53.927029 28026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b01a4f7ef5461a956c288f9a9458d0" candidate_uuid: "d85cddb1910a4ca980af31262d4474dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fe00ee12f19c43b3aef56af3b569ee00"
I20240503 02:18:53.927263 28026 raft_consensus.cc:3032] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:18:53.927342 28026 raft_consensus.cc:729] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe00ee12f19c43b3aef56af3b569ee00, State: Running, Role: LEADER
I20240503 02:18:53.927645 28026 consensus_queue.cc:260] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } }
I20240503 02:18:53.927959 28026 raft_consensus.cc:3037] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:18:53.930578 28026 raft_consensus.cc:2454] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d85cddb1910a4ca980af31262d4474dd in term 2.
I20240503 02:18:53.931201 28539 leader_election.cc:302] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d85cddb1910a4ca980af31262d4474dd, fe00ee12f19c43b3aef56af3b569ee00; no voters: 
I20240503 02:18:53.931538 29006 raft_consensus.cc:2781] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:18:53.931689 29006 raft_consensus.cc:686] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [term 2 LEADER]: Becoming Leader. State: Replica: d85cddb1910a4ca980af31262d4474dd, State: Running, Role: LEADER
I20240503 02:18:53.931960 29006 consensus_queue.cc:237] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } }
I20240503 02:18:53.934459 27752 catalog_manager.cc:5484] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd reported cstate change: term changed from 1 to 2, leader changed from fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2) to d85cddb1910a4ca980af31262d4474dd (127.0.145.3). New cstate: current_term: 2 leader_uuid: "d85cddb1910a4ca980af31262d4474dd" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:54.054013 27895 raft_consensus.cc:1261] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 2 FOLLOWER]: Refusing update from remote peer 57d37d4403774fb8b57c63a6d53f7dff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:18:54.054891 29005 consensus_queue.cc:1035] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:18:54.070129 28026 raft_consensus.cc:1261] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 2 FOLLOWER]: Refusing update from remote peer 57d37d4403774fb8b57c63a6d53f7dff: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:18:54.070845 29190 consensus_queue.cc:1035] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:18:54.073403 28745 consensus_queue.cc:237] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:54.074199 29005 raft_consensus.cc:2932] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5b5ff5d736d344869048c4155e43b56a (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } } }
I20240503 02:18:54.077651 27741 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e833cff4caf947a986d4abb1df7ef4fb with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 5)
I20240503 02:18:54.079633 27752 catalog_manager.cc:5484] T e833cff4caf947a986d4abb1df7ef4fb P 57d37d4403774fb8b57c63a6d53f7dff reported cstate change: config changed from index 4 to 6, VOTER 5b5ff5d736d344869048c4155e43b56a (127.0.145.1) evicted. New cstate: current_term: 2 leader_uuid: "57d37d4403774fb8b57c63a6d53f7dff" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:54.082994 27875 tablet_service.cc:1514] Processing DeleteTablet for tablet e833cff4caf947a986d4abb1df7ef4fb with delete_type TABLET_DATA_TOMBSTONED (TS 5b5ff5d736d344869048c4155e43b56a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37338
I20240503 02:18:54.083770 29264 tablet_replica.cc:330] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a: stopping tablet replica
I20240503 02:18:54.084177 29264 raft_consensus.cc:2229] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:18:54.084425 29264 raft_consensus.cc:2258] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:18:54.085621 29264 ts_tablet_manager.cc:1875] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:18:54.090246 29264 ts_tablet_manager.cc:1888] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:18:54.090436 29264 log.cc:1196] T e833cff4caf947a986d4abb1df7ef4fb P 5b5ff5d736d344869048c4155e43b56a: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/e833cff4caf947a986d4abb1df7ef4fb
I20240503 02:18:54.091363 27740 catalog_manager.cc:4830] TS 5b5ff5d736d344869048c4155e43b56a (127.0.145.1:38711): tablet e833cff4caf947a986d4abb1df7ef4fb (table rebalance_test_table_0 [id=dc591e3b8e6e4667b50bf68f42852d13]) successfully deleted
I20240503 02:18:54.149344 28026 raft_consensus.cc:1261] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 2 FOLLOWER]: Refusing update from remote peer 57d37d4403774fb8b57c63a6d53f7dff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:18:54.149297 28745 consensus_queue.cc:237] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:54.149999 29005 consensus_queue.cc:1035] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:18:54.150153 29262 raft_consensus.cc:2932] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } } }
I20240503 02:18:54.154565 27741 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7875d314f4a54e9cb2790b6a31f1ee8c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:18:54.156260 27752 catalog_manager.cc:5484] T 7875d314f4a54e9cb2790b6a31f1ee8c P 57d37d4403774fb8b57c63a6d53f7dff reported cstate change: config changed from index 3 to 5, VOTER fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2) evicted. New cstate: current_term: 2 leader_uuid: "57d37d4403774fb8b57c63a6d53f7dff" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:54.159451 28006 tablet_service.cc:1514] Processing DeleteTablet for tablet 7875d314f4a54e9cb2790b6a31f1ee8c with delete_type TABLET_DATA_TOMBSTONED (TS fe00ee12f19c43b3aef56af3b569ee00 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57738
I20240503 02:18:54.160557 29265 tablet_replica.cc:330] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00: stopping tablet replica
I20240503 02:18:54.161120 29265 raft_consensus.cc:2229] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:18:54.161428 29265 raft_consensus.cc:2258] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:18:54.162940 29265 ts_tablet_manager.cc:1875] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:18:54.167384 29265 ts_tablet_manager.cc:1888] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 02:18:54.167516 29265 log.cc:1196] T 7875d314f4a54e9cb2790b6a31f1ee8c P fe00ee12f19c43b3aef56af3b569ee00: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/7875d314f4a54e9cb2790b6a31f1ee8c
I20240503 02:18:54.168359 27740 catalog_manager.cc:4830] TS fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2:35911): tablet 7875d314f4a54e9cb2790b6a31f1ee8c (table rebalance_test_table_1 [id=2b1c093c7c854f608a38cea5c1d13449]) successfully deleted
I20240503 02:18:54.234964 27895 raft_consensus.cc:1261] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 2 FOLLOWER]: Refusing update from remote peer d85cddb1910a4ca980af31262d4474dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:18:54.235625 29006 consensus_queue.cc:1035] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b5ff5d736d344869048c4155e43b56a" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 38711 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:18:54.264145 28606 consensus_queue.cc:237] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } }
I20240503 02:18:54.265213 29193 raft_consensus.cc:2932] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5b5ff5d736d344869048c4155e43b56a (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } } }
I20240503 02:18:54.269955 27738 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56b204d83cb148dc841afb1e2aaedbba with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:18:54.270957 27750 catalog_manager.cc:5484] T 56b204d83cb148dc841afb1e2aaedbba P d85cddb1910a4ca980af31262d4474dd reported cstate change: config changed from index 4 to 6, VOTER 5b5ff5d736d344869048c4155e43b56a (127.0.145.1) evicted. New cstate: current_term: 2 leader_uuid: "d85cddb1910a4ca980af31262d4474dd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:54.274307 27875 tablet_service.cc:1514] Processing DeleteTablet for tablet 56b204d83cb148dc841afb1e2aaedbba with delete_type TABLET_DATA_TOMBSTONED (TS 5b5ff5d736d344869048c4155e43b56a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37338
I20240503 02:18:54.274662 29264 tablet_replica.cc:330] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a: stopping tablet replica
I20240503 02:18:54.274856 29264 raft_consensus.cc:2229] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:18:54.275033 29264 raft_consensus.cc:2258] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:18:54.275775 29264 ts_tablet_manager.cc:1875] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:18:54.279911 29264 ts_tablet_manager.cc:1888] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:18:54.280015 29264 log.cc:1196] T 56b204d83cb148dc841afb1e2aaedbba P 5b5ff5d736d344869048c4155e43b56a: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/56b204d83cb148dc841afb1e2aaedbba
I20240503 02:18:54.280647 27740 catalog_manager.cc:4830] TS 5b5ff5d736d344869048c4155e43b56a (127.0.145.1:38711): tablet 56b204d83cb148dc841afb1e2aaedbba (table rebalance_test_table_2 [id=50e4a3d399854e64b553e1074d4529b0]) successfully deleted
I20240503 02:18:54.348516 28745 consensus_queue.cc:237] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } }
I20240503 02:18:54.349398 29262 raft_consensus.cc:2932] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } } }
I20240503 02:18:54.352058 27741 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d8349bd8ed5d49d4a4f6cd99239b1ebc with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:18:54.353415 27750 catalog_manager.cc:5484] T d8349bd8ed5d49d4a4f6cd99239b1ebc P 57d37d4403774fb8b57c63a6d53f7dff reported cstate change: config changed from index 4 to 6, VOTER fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2) evicted. New cstate: current_term: 2 leader_uuid: "57d37d4403774fb8b57c63a6d53f7dff" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57d37d4403774fb8b57c63a6d53f7dff" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 40035 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:54.356567 28006 tablet_service.cc:1514] Processing DeleteTablet for tablet d8349bd8ed5d49d4a4f6cd99239b1ebc with delete_type TABLET_DATA_TOMBSTONED (TS fe00ee12f19c43b3aef56af3b569ee00 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57738
I20240503 02:18:54.356889 29265 tablet_replica.cc:330] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00: stopping tablet replica
I20240503 02:18:54.357041 29265 raft_consensus.cc:2229] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:18:54.357199 29265 raft_consensus.cc:2258] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:18:54.357867 29265 ts_tablet_manager.cc:1875] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:18:54.362552 29265 ts_tablet_manager.cc:1888] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:18:54.362721 29265 log.cc:1196] T d8349bd8ed5d49d4a4f6cd99239b1ebc P fe00ee12f19c43b3aef56af3b569ee00: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/d8349bd8ed5d49d4a4f6cd99239b1ebc
I20240503 02:18:54.363400 27740 catalog_manager.cc:4830] TS fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2:35911): tablet d8349bd8ed5d49d4a4f6cd99239b1ebc (table rebalance_test_table_3 [id=bfa3aab996bd4ee4a61113cff9ce2cb1]) successfully deleted
I20240503 02:18:54.455708 28026 raft_consensus.cc:1261] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 2 FOLLOWER]: Refusing update from remote peer d85cddb1910a4ca980af31262d4474dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:18:54.456259 29193 consensus_queue.cc:1035] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe00ee12f19c43b3aef56af3b569ee00" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 35911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:18:54.557709 28605 consensus_queue.cc:237] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } }
I20240503 02:18:54.558629 29193 raft_consensus.cc:2932] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } } }
I20240503 02:18:54.561297 27738 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 04b01a4f7ef5461a956c288f9a9458d0 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:18:54.562629 27752 catalog_manager.cc:5484] T 04b01a4f7ef5461a956c288f9a9458d0 P d85cddb1910a4ca980af31262d4474dd reported cstate change: config changed from index 4 to 6, VOTER fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2) evicted. New cstate: current_term: 2 leader_uuid: "d85cddb1910a4ca980af31262d4474dd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d85cddb1910a4ca980af31262d4474dd" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 36141 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:18:54.565860 28006 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b01a4f7ef5461a956c288f9a9458d0 with delete_type TABLET_DATA_TOMBSTONED (TS fe00ee12f19c43b3aef56af3b569ee00 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57738
I20240503 02:18:54.566167 29265 tablet_replica.cc:330] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00: stopping tablet replica
I20240503 02:18:54.566321 29265 raft_consensus.cc:2229] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:18:54.566470 29265 raft_consensus.cc:2258] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:18:54.567185 29265 ts_tablet_manager.cc:1875] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:18:54.571663 29265 ts_tablet_manager.cc:1888] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:18:54.571774 29265 log.cc:1196] T 04b01a4f7ef5461a956c288f9a9458d0 P fe00ee12f19c43b3aef56af3b569ee00: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/04b01a4f7ef5461a956c288f9a9458d0
I20240503 02:18:54.572423 27740 catalog_manager.cc:4830] TS fe00ee12f19c43b3aef56af3b569ee00 (127.0.145.2:35911): tablet 04b01a4f7ef5461a956c288f9a9458d0 (table rebalance_test_table_1 [id=2b1c093c7c854f608a38cea5c1d13449]) successfully deleted
I20240503 02:18:54.815135 28725 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:54.823889 28586 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:54.830508 27875 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:18:54.830672 28006 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:54.976174 28725 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:54.985730 28586 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:54.986202 27875 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:18:55.004431 28006 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:55.262499 28725 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:55.267606 28006 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:55.275923 28586 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:55.291843 27875 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
               UUID               |      Address       | Status
----------------------------------+--------------------+---------
 1e862dc3b1c34c47b24dd8f646cb6f8e | 127.0.145.62:43077 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                              Value                                                                                               |         Tags          |         Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                                              | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                                              | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                                             | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                                | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                                             | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                                             | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                             | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                               | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                                              | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.0.145.20:37283 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address      | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
 57d37d4403774fb8b57c63a6d53f7dff | 127.0.145.4:40035 | HEALTHY | <none>   |       3        |       0
 5b5ff5d736d344869048c4155e43b56a | 127.0.145.1:38711 | HEALTHY | <none>   |       4        |       0
 d85cddb1910a4ca980af31262d4474dd | 127.0.145.3:36141 | HEALTHY | <none>   |       3        |       0
 fe00ee12f19c43b3aef56af3b569ee00 | 127.0.145.2:35911 | HEALTHY | <none>   |       3        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
                   Flag                   |                                                                                            Value                                                                                             |         Tags          |      Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                                          | experimental          | all 4 server(s) checked
 local_ip_for_outbound_sockets            | 127.0.145.1                                                                                                                                                                                  | experimental          | 127.0.145.1:38711
 local_ip_for_outbound_sockets            | 127.0.145.2                                                                                                                                                                                  | experimental          | 127.0.145.2:35911
 local_ip_for_outbound_sockets            | 127.0.145.3                                                                                                                                                                                  | experimental          | 127.0.145.3:36141
 local_ip_for_outbound_sockets            | 127.0.145.4                                                                                                                                                                                  | experimental          | 127.0.145.4:40035
 never_fsync                              | true                                                                                                                                                                                         | unsafe,advanced       | all 4 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                            | unsafe,hidden         | all 4 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                                         | experimental,advanced | all 4 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                         | unsafe                | all 4 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                           | hidden                | all 4 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.0.145.1:38711
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.0.145.2:35911
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.0.145.3:36141
 server_dump_info_path                    | /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.0.145.4:40035

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.0.145.20:37283 | all 4 server(s) checked
 time_source         | builtin            | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 1  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 3
 First Quartile | 3
 Median         | 3
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 5
 Tablets        | 13
 Replicas       | 13

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240503 02:18:55.332513   580 log_verifier.cc:126] Checking tablet 04b01a4f7ef5461a956c288f9a9458d0
I20240503 02:18:55.341171   580 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 04b01a4f7ef5461a956c288f9a9458d0
I20240503 02:18:55.341277   580 log_verifier.cc:126] Checking tablet 117bdd53a4784124a5fc9704b0755144
I20240503 02:18:55.348650   580 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 117bdd53a4784124a5fc9704b0755144
I20240503 02:18:55.348752   580 log_verifier.cc:126] Checking tablet 189a5b9a4d2943548d9fbcd25e6cf0f7
I20240503 02:18:55.383124   580 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 189a5b9a4d2943548d9fbcd25e6cf0f7
I20240503 02:18:55.383222   580 log_verifier.cc:126] Checking tablet 56b204d83cb148dc841afb1e2aaedbba
I20240503 02:18:55.391711   580 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 56b204d83cb148dc841afb1e2aaedbba
I20240503 02:18:55.391844   580 log_verifier.cc:126] Checking tablet 6096f9bd19e749889c8e6154d88c921d
I20240503 02:18:55.399376   580 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 6096f9bd19e749889c8e6154d88c921d
I20240503 02:18:55.399464   580 log_verifier.cc:126] Checking tablet 6a6678b8bc4d4867a39bf894d2f8d0f3
I20240503 02:18:55.407050   580 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 6a6678b8bc4d4867a39bf894d2f8d0f3
I20240503 02:18:55.407137   580 log_verifier.cc:126] Checking tablet 7875d314f4a54e9cb2790b6a31f1ee8c
I20240503 02:18:55.415431   580 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 7875d314f4a54e9cb2790b6a31f1ee8c
I20240503 02:18:55.415546   580 log_verifier.cc:126] Checking tablet a8c411abd9f14362ae6a4efbdca4ac10
I20240503 02:18:55.423305   580 log_verifier.cc:177] Verified matching terms for 2 ops in tablet a8c411abd9f14362ae6a4efbdca4ac10
I20240503 02:18:55.423413   580 log_verifier.cc:126] Checking tablet b1a551ede9bb43c48b062df3c404bdd1
I20240503 02:18:55.431100   580 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b1a551ede9bb43c48b062df3c404bdd1
I20240503 02:18:55.431212   580 log_verifier.cc:126] Checking tablet b9e2225705cb41dbaeccbe83e1dc03d1
I20240503 02:18:55.439028   580 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b9e2225705cb41dbaeccbe83e1dc03d1
I20240503 02:18:55.439136   580 log_verifier.cc:126] Checking tablet d8349bd8ed5d49d4a4f6cd99239b1ebc
I20240503 02:18:55.447759   580 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d8349bd8ed5d49d4a4f6cd99239b1ebc
I20240503 02:18:55.447889   580 log_verifier.cc:126] Checking tablet e65563b9a9314a6a89ee51d006513dad
I20240503 02:18:55.456012   580 log_verifier.cc:177] Verified matching terms for 1 ops in tablet e65563b9a9314a6a89ee51d006513dad
I20240503 02:18:55.456117   580 log_verifier.cc:126] Checking tablet e833cff4caf947a986d4abb1df7ef4fb
I20240503 02:18:55.465121   580 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e833cff4caf947a986d4abb1df7ef4fb
I20240503 02:18:55.466248   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 27813
I20240503 02:18:55.467700 27934 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:18:55.578217   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 27813
I20240503 02:18:55.600170   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 27944
I20240503 02:18:55.601222 28066 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:18:55.715531   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 27944
I20240503 02:18:55.743925   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 28522
I20240503 02:18:55.750305 28651 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:18:55.853813   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 28522
I20240503 02:18:55.874063   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 28660
I20240503 02:18:55.880026 28785 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:18:55.986889   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 28660
I20240503 02:18:56.006364   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 27723
I20240503 02:18:56.007508 27784 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:18:56.127112   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 27723
2024-05-03T02:18:56Z chronyd exiting
[       OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1 (16074 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (16074 ms total)

[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN      ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3
I20240503 02:18:56.185271   580 test_util.cc:268] Using random seed: 1467766686
I20240503 02:18:56.187103   580 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:18:56.187173   580 ts_itest-base.cc:113] --------------
I20240503 02:18:56.187220   580 ts_itest-base.cc:114] 6 tablet servers
I20240503 02:18:56.187274   580 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:18:56.187328   580 ts_itest-base.cc:116] --------------
2024-05-03T02:18:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:18:56Z Disabled control of system clock
I20240503 02:18:56.236454   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.145.62:44317
--webserver_interface=127.0.145.62
--webserver_port=0
--builtin_ntp_servers=127.0.145.20:46749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.145.62:44317
--location_mapping_cmd=/tmp/dist-test-taskfpuZdt/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 02:18:56.574069 29434 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 02:18:56.574473 29434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:18:56.574590 29434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:18:56.574756 29434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:18:56.582432 29434 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:18:56.582597 29434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:18:56.582737 29434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:18:56.582806 29434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:18:56.582895 29434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:18:56.592381 29434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:46749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskfpuZdt/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--ipki_ca_key_size=768
--master_addresses=127.0.145.62:44317
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.145.62:44317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.145.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:18:56.593780 29434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:18:56.595669 29434 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:18:56.607161 29439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:56.607687 29442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:56.608397 29440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:18:56.609210 29434 server_base.cc:1034] running on GCE node
I20240503 02:18:56.609939 29434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:18:56.611399 29434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:18:56.612602 29434 hybrid_clock.cc:648] HybridClock initialized: now 1714702736612552 us; error 40 us; skew 500 ppm
I20240503 02:18:56.612900 29434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:18:56.615139 29434 webserver.cc:460] Webserver started at http://127.0.145.62:35153/ using document root <none> and password file <none>
I20240503 02:18:56.615844 29434 fs_manager.cc:359] Metadata directory not provided
I20240503 02:18:56.615968 29434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:18:56.616304 29434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:18:56.618989 29434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c35820a8e798485185efe33885a7403c"
format_stamp: "Formatted at 2024-05-03 02:18:56 on dist-test-slave-gq9h"
I20240503 02:18:56.619771 29434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c35820a8e798485185efe33885a7403c"
format_stamp: "Formatted at 2024-05-03 02:18:56 on dist-test-slave-gq9h"
I20240503 02:18:56.624958 29434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:18:56.628435 29448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:56.629720 29434 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240503 02:18:56.629890 29434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c35820a8e798485185efe33885a7403c"
format_stamp: "Formatted at 2024-05-03 02:18:56 on dist-test-slave-gq9h"
I20240503 02:18:56.630057 29434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:18:56.655485 29434 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:18:56.656358 29434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:18:56.656644 29434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:18:56.676328 29434 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.62:44317
I20240503 02:18:56.676345 29499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.62:44317 every 8 connection(s)
I20240503 02:18:56.677767 29434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:18:56.681767 29500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:18:56.687896   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 29434
I20240503 02:18:56.688122   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:18:56.692750 29500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c: Bootstrap starting.
I20240503 02:18:56.696868 29500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c: Neither blocks nor log segments found. Creating new log.
I20240503 02:18:56.698326 29500 log.cc:824] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c: Log is configured to *not* fsync() on all Append() calls
I20240503 02:18:56.701694 29500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c: No bootstrap required, opened a new log
I20240503 02:18:56.708251 29500 raft_consensus.cc:348] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35820a8e798485185efe33885a7403c" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 44317 } }
I20240503 02:18:56.708613 29500 raft_consensus.cc:374] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:18:56.708686 29500 raft_consensus.cc:729] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35820a8e798485185efe33885a7403c, State: Initialized, Role: FOLLOWER
I20240503 02:18:56.709255 29500 consensus_queue.cc:260] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35820a8e798485185efe33885a7403c" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 44317 } }
I20240503 02:18:56.709439 29500 raft_consensus.cc:388] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:18:56.709530 29500 raft_consensus.cc:482] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:18:56.709643 29500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:18:56.711728 29500 raft_consensus.cc:504] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35820a8e798485185efe33885a7403c" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 44317 } }
I20240503 02:18:56.712255 29500 leader_election.cc:302] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c35820a8e798485185efe33885a7403c; no voters: 
I20240503 02:18:56.712894 29500 leader_election.cc:288] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:18:56.713060 29505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:18:56.713740 29505 raft_consensus.cc:686] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [term 1 LEADER]: Becoming Leader. State: Replica: c35820a8e798485185efe33885a7403c, State: Running, Role: LEADER
I20240503 02:18:56.714296 29505 consensus_queue.cc:237] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35820a8e798485185efe33885a7403c" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 44317 } }
I20240503 02:18:56.715046 29500 sys_catalog.cc:564] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:18:56.717144 29507 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c35820a8e798485185efe33885a7403c. Latest consensus state: current_term: 1 leader_uuid: "c35820a8e798485185efe33885a7403c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35820a8e798485185efe33885a7403c" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 44317 } } }
I20240503 02:18:56.717396 29507 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [sys.catalog]: This master's current role is: LEADER
I20240503 02:18:56.717778 29506 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c35820a8e798485185efe33885a7403c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35820a8e798485185efe33885a7403c" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 44317 } } }
I20240503 02:18:56.718047 29506 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c [sys.catalog]: This master's current role is: LEADER
I20240503 02:18:56.718837 29511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:18:56.723834 29511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:18:56.732403 29511 catalog_manager.cc:1347] Generated new cluster ID: 8c92f2259545409ea7f98016cde0aa65
I20240503 02:18:56.732533 29511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:18:56.740121 29511 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:18:56.741838 29511 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:18:56.756232 29511 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c35820a8e798485185efe33885a7403c: Generated new TSK 0
I20240503 02:18:56.757349 29511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:18:56.764063   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.1:0
--local_ip_for_outbound_sockets=127.0.145.1
--webserver_interface=127.0.145.1
--webserver_port=0
--tserver_master_addrs=127.0.145.62:44317
--builtin_ntp_servers=127.0.145.20:46749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:18:57.106714 29524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:18:57.107152 29524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:18:57.107296 29524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:18:57.114884 29524 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:18:57.115057 29524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:18:57.115238 29524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.1
I20240503 02:18:57.124209 29524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:46749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.145.1
--webserver_port=0
--tserver_master_addrs=127.0.145.62:44317
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.1
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:18:57.125612 29524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:18:57.127439 29524 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:18:57.140239 29532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:57.140810 29529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:57.141512 29530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:18:57.142168 29524 server_base.cc:1034] running on GCE node
I20240503 02:18:57.142858 29524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:18:57.143801 29524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:18:57.145010 29524 hybrid_clock.cc:648] HybridClock initialized: now 1714702737144963 us; error 39 us; skew 500 ppm
I20240503 02:18:57.145334 29524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:18:57.147758 29524 webserver.cc:460] Webserver started at http://127.0.145.1:43079/ using document root <none> and password file <none>
I20240503 02:18:57.148526 29524 fs_manager.cc:359] Metadata directory not provided
I20240503 02:18:57.148644 29524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:18:57.148972 29524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:18:57.151692 29524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "07bd09ad30cc4932bfff380344c8e9fc"
format_stamp: "Formatted at 2024-05-03 02:18:57 on dist-test-slave-gq9h"
I20240503 02:18:57.152540 29524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "07bd09ad30cc4932bfff380344c8e9fc"
format_stamp: "Formatted at 2024-05-03 02:18:57 on dist-test-slave-gq9h"
I20240503 02:18:57.157840 29524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.002s
I20240503 02:18:57.161345 29538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:57.162688 29524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:18:57.162878 29524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "07bd09ad30cc4932bfff380344c8e9fc"
format_stamp: "Formatted at 2024-05-03 02:18:57 on dist-test-slave-gq9h"
I20240503 02:18:57.163081 29524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:18:57.185627 29524 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:18:57.186478 29524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:18:57.186764 29524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:18:57.187716 29524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:18:57.189262 29524 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:18:57.189350 29524 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:57.189469 29524 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:18:57.189553 29524 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:57.221501 29524 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.1:46557
I20240503 02:18:57.221554 29650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.1:46557 every 8 connection(s)
I20240503 02:18:57.222829 29524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:18:57.224148   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 29524
I20240503 02:18:57.224361   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:18:57.228271   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.2:0
--local_ip_for_outbound_sockets=127.0.145.2
--webserver_interface=127.0.145.2
--webserver_port=0
--tserver_master_addrs=127.0.145.62:44317
--builtin_ntp_servers=127.0.145.20:46749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:18:57.237525 29651 heartbeater.cc:347] Connected to a master server at 127.0.145.62:44317
I20240503 02:18:57.237890 29651 heartbeater.cc:464] Registering TS with master...
I20240503 02:18:57.238947 29651 heartbeater.cc:511] Master 127.0.145.62:44317 requested a full tablet report, sending...
I20240503 02:18:57.285243 29465 ts_manager.cc:194] Registered new tserver with Master: 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557)
I20240503 02:18:57.287798 29465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.1:60525
W20240503 02:18:57.568637 29655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:18:57.568980 29655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:18:57.569085 29655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:18:57.576357 29655 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:18:57.576500 29655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:18:57.576627 29655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.2
I20240503 02:18:57.585630 29655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:46749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.145.2
--webserver_port=0
--tserver_master_addrs=127.0.145.62:44317
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.2
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:18:57.587250 29655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:18:57.589126 29655 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:18:57.601908 29661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:57.602675 29662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:57.603085 29664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:18:57.604735 29655 server_base.cc:1034] running on GCE node
I20240503 02:18:57.605360 29655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:18:57.606299 29655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:18:57.607498 29655 hybrid_clock.cc:648] HybridClock initialized: now 1714702737607452 us; error 46 us; skew 500 ppm
I20240503 02:18:57.607826 29655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:18:57.615162 29655 webserver.cc:460] Webserver started at http://127.0.145.2:44575/ using document root <none> and password file <none>
I20240503 02:18:57.615993 29655 fs_manager.cc:359] Metadata directory not provided
I20240503 02:18:57.616117 29655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:18:57.616470 29655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:18:57.619199 29655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "99f937fbfb7c49f2afe278eaa73edbba"
format_stamp: "Formatted at 2024-05-03 02:18:57 on dist-test-slave-gq9h"
I20240503 02:18:57.620057 29655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "99f937fbfb7c49f2afe278eaa73edbba"
format_stamp: "Formatted at 2024-05-03 02:18:57 on dist-test-slave-gq9h"
I20240503 02:18:57.625329 29655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:18:57.628772 29670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:57.630122 29655 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:18:57.630281 29655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "99f937fbfb7c49f2afe278eaa73edbba"
format_stamp: "Formatted at 2024-05-03 02:18:57 on dist-test-slave-gq9h"
I20240503 02:18:57.630493 29655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:18:57.649006 29655 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:18:57.649972 29655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:18:57.650220 29655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:18:57.651240 29655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:18:57.652712 29655 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:18:57.652799 29655 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:57.652884 29655 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:18:57.652942 29655 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:57.690326 29655 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.2:37371
I20240503 02:18:57.690400 29782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.2:37371 every 8 connection(s)
I20240503 02:18:57.691720 29655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:18:57.693959   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 29655
I20240503 02:18:57.694139   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:18:57.698164   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.3:0
--local_ip_for_outbound_sockets=127.0.145.3
--webserver_interface=127.0.145.3
--webserver_port=0
--tserver_master_addrs=127.0.145.62:44317
--builtin_ntp_servers=127.0.145.20:46749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:18:57.707664 29783 heartbeater.cc:347] Connected to a master server at 127.0.145.62:44317
I20240503 02:18:57.707981 29783 heartbeater.cc:464] Registering TS with master...
I20240503 02:18:57.708843 29783 heartbeater.cc:511] Master 127.0.145.62:44317 requested a full tablet report, sending...
I20240503 02:18:57.770241 29465 ts_manager.cc:194] Registered new tserver with Master: 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371)
I20240503 02:18:57.771204 29465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.2:41397
W20240503 02:18:58.060385 29786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:18:58.060688 29786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:18:58.060807 29786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:18:58.068176 29786 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:18:58.068320 29786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:18:58.068465 29786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.3
I20240503 02:18:58.077383 29786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:46749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.145.3
--webserver_port=0
--tserver_master_addrs=127.0.145.62:44317
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.3
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:18:58.078719 29786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:18:58.080410 29786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:18:58.092718 29792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:58.093353 29795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:58.094022 29793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:18:58.094666 29786 server_base.cc:1034] running on GCE node
I20240503 02:18:58.095274 29786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:18:58.096230 29786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:18:58.097434 29786 hybrid_clock.cc:648] HybridClock initialized: now 1714702738097385 us; error 43 us; skew 500 ppm
I20240503 02:18:58.097748 29786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:18:58.100068 29786 webserver.cc:460] Webserver started at http://127.0.145.3:41233/ using document root <none> and password file <none>
I20240503 02:18:58.100817 29786 fs_manager.cc:359] Metadata directory not provided
I20240503 02:18:58.100935 29786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:18:58.101258 29786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:18:58.103858 29786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
format_stamp: "Formatted at 2024-05-03 02:18:58 on dist-test-slave-gq9h"
I20240503 02:18:58.104619 29786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
format_stamp: "Formatted at 2024-05-03 02:18:58 on dist-test-slave-gq9h"
I20240503 02:18:58.109925 29786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.004s
I20240503 02:18:58.113364 29801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:58.114769 29786 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240503 02:18:58.114926 29786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
format_stamp: "Formatted at 2024-05-03 02:18:58 on dist-test-slave-gq9h"
I20240503 02:18:58.115128 29786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:18:58.132156 29786 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:18:58.133046 29786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:18:58.133324 29786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:18:58.134370 29786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:18:58.135983 29786 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:18:58.136085 29786 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:58.136201 29786 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:18:58.136276 29786 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:58.168577 29786 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.3:41133
I20240503 02:18:58.168634 29913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.3:41133 every 8 connection(s)
I20240503 02:18:58.169920 29786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:18:58.175770   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 29786
I20240503 02:18:58.176028   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:18:58.179775   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.4:0
--local_ip_for_outbound_sockets=127.0.145.4
--webserver_interface=127.0.145.4
--webserver_port=0
--tserver_master_addrs=127.0.145.62:44317
--builtin_ntp_servers=127.0.145.20:46749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:18:58.184327 29914 heartbeater.cc:347] Connected to a master server at 127.0.145.62:44317
I20240503 02:18:58.184645 29914 heartbeater.cc:464] Registering TS with master...
I20240503 02:18:58.187337 29914 heartbeater.cc:511] Master 127.0.145.62:44317 requested a full tablet report, sending...
I20240503 02:18:58.233741 29465 ts_manager.cc:194] Registered new tserver with Master: 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:18:58.235209 29465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.3:50453
I20240503 02:18:58.291139 29651 heartbeater.cc:503] Master 127.0.145.62:44317 was elected leader, sending a full tablet report...
W20240503 02:18:58.521835 29917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:18:58.522152 29917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:18:58.522302 29917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:18:58.529219 29917 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:18:58.529397 29917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:18:58.529565 29917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.4
I20240503 02:18:58.538429 29917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:46749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.145.4
--webserver_port=0
--tserver_master_addrs=127.0.145.62:44317
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.4
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:18:58.539815 29917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:18:58.541581 29917 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:18:58.553537 29923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:58.554024 29926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:58.554519 29924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:18:58.555424 29917 server_base.cc:1034] running on GCE node
I20240503 02:18:58.556046 29917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:18:58.556999 29917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:18:58.558223 29917 hybrid_clock.cc:648] HybridClock initialized: now 1714702738558159 us; error 60 us; skew 500 ppm
I20240503 02:18:58.558537 29917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:18:58.560832 29917 webserver.cc:460] Webserver started at http://127.0.145.4:42005/ using document root <none> and password file <none>
I20240503 02:18:58.561563 29917 fs_manager.cc:359] Metadata directory not provided
I20240503 02:18:58.561694 29917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:18:58.562007 29917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:18:58.564559 29917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "319ccec294484fe9ac61b2e718311b4c"
format_stamp: "Formatted at 2024-05-03 02:18:58 on dist-test-slave-gq9h"
I20240503 02:18:58.565377 29917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "319ccec294484fe9ac61b2e718311b4c"
format_stamp: "Formatted at 2024-05-03 02:18:58 on dist-test-slave-gq9h"
I20240503 02:18:58.570540 29917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240503 02:18:58.573836 29932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:58.575228 29917 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:18:58.575387 29917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "319ccec294484fe9ac61b2e718311b4c"
format_stamp: "Formatted at 2024-05-03 02:18:58 on dist-test-slave-gq9h"
I20240503 02:18:58.575582 29917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:18:58.600828 29917 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:18:58.601714 29917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:18:58.601977 29917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:18:58.603040 29917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:18:58.604504 29917 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:18:58.604621 29917 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:58.604732 29917 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:18:58.604800 29917 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:58.637243 29917 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.4:39727
I20240503 02:18:58.637302 30044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.4:39727 every 8 connection(s)
I20240503 02:18:58.638556 29917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:18:58.643905   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 29917
I20240503 02:18:58.644055   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:18:58.647854   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.5:0
--local_ip_for_outbound_sockets=127.0.145.5
--webserver_interface=127.0.145.5
--webserver_port=0
--tserver_master_addrs=127.0.145.62:44317
--builtin_ntp_servers=127.0.145.20:46749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:18:58.651765 30045 heartbeater.cc:347] Connected to a master server at 127.0.145.62:44317
I20240503 02:18:58.652076 30045 heartbeater.cc:464] Registering TS with master...
I20240503 02:18:58.653015 30045 heartbeater.cc:511] Master 127.0.145.62:44317 requested a full tablet report, sending...
I20240503 02:18:58.701198 29465 ts_manager.cc:194] Registered new tserver with Master: 319ccec294484fe9ac61b2e718311b4c (127.0.145.4:39727)
I20240503 02:18:58.702136 29465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.4:59519
I20240503 02:18:58.774291 29783 heartbeater.cc:503] Master 127.0.145.62:44317 was elected leader, sending a full tablet report...
W20240503 02:18:58.995848 30048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:18:58.996312 30048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:18:58.996510 30048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:18:59.004339 30048 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:18:59.004477 30048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:18:59.004642 30048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.5
I20240503 02:18:59.013886 30048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:46749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.145.5
--webserver_port=0
--tserver_master_addrs=127.0.145.62:44317
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.5
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:18:59.015223 30048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:18:59.016877 30048 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:18:59.029029 30054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:59.029526 30057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:59.030007 30055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:18:59.030906 30048 server_base.cc:1034] running on GCE node
I20240503 02:18:59.031522 30048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:18:59.032439 30048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:18:59.033666 30048 hybrid_clock.cc:648] HybridClock initialized: now 1714702739033606 us; error 39 us; skew 500 ppm
I20240503 02:18:59.033999 30048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:18:59.036458 30048 webserver.cc:460] Webserver started at http://127.0.145.5:46513/ using document root <none> and password file <none>
I20240503 02:18:59.037243 30048 fs_manager.cc:359] Metadata directory not provided
I20240503 02:18:59.037359 30048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:18:59.037667 30048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:18:59.040333 30048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "209bba2a25244371943e842bdf9267cd"
format_stamp: "Formatted at 2024-05-03 02:18:59 on dist-test-slave-gq9h"
I20240503 02:18:59.041123 30048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "209bba2a25244371943e842bdf9267cd"
format_stamp: "Formatted at 2024-05-03 02:18:59 on dist-test-slave-gq9h"
I20240503 02:18:59.046494 30048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.005s
I20240503 02:18:59.050149 30063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:59.051674 30048 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240503 02:18:59.051846 30048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "209bba2a25244371943e842bdf9267cd"
format_stamp: "Formatted at 2024-05-03 02:18:59 on dist-test-slave-gq9h"
I20240503 02:18:59.052067 30048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:18:59.066200 30048 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:18:59.067188 30048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:18:59.067449 30048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:18:59.068447 30048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:18:59.069849 30048 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:18:59.069940 30048 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:59.070021 30048 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:18:59.070077 30048 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:59.101836 30048 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.5:35793
I20240503 02:18:59.101909 30175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.5:35793 every 8 connection(s)
I20240503 02:18:59.103168 30048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:18:59.112160   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 30048
I20240503 02:18:59.112361   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:18:59.115921 30176 heartbeater.cc:347] Connected to a master server at 127.0.145.62:44317
I20240503 02:18:59.115936   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.6:0
--local_ip_for_outbound_sockets=127.0.145.6
--webserver_interface=127.0.145.6
--webserver_port=0
--tserver_master_addrs=127.0.145.62:44317
--builtin_ntp_servers=127.0.145.20:46749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:18:59.116261 30176 heartbeater.cc:464] Registering TS with master...
I20240503 02:18:59.117079 30176 heartbeater.cc:511] Master 127.0.145.62:44317 requested a full tablet report, sending...
I20240503 02:18:59.163987 29465 ts_manager.cc:194] Registered new tserver with Master: 209bba2a25244371943e842bdf9267cd (127.0.145.5:35793)
I20240503 02:18:59.164932 29465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.5:40367
I20240503 02:18:59.237888 29914 heartbeater.cc:503] Master 127.0.145.62:44317 was elected leader, sending a full tablet report...
W20240503 02:18:59.459165 30180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:18:59.459478 30180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:18:59.459627 30180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:18:59.467105 30180 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:18:59.467239 30180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:18:59.467396 30180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.6
I20240503 02:18:59.476411 30180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:46749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.145.6
--webserver_port=0
--tserver_master_addrs=127.0.145.62:44317
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.6
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:18:59.477780 30180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:18:59.479632 30180 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:18:59.492352 30186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:59.492794 30188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:18:59.493568 30185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:18:59.494347 30180 server_base.cc:1034] running on GCE node
I20240503 02:18:59.495100 30180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:18:59.496074 30180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:18:59.497278 30180 hybrid_clock.cc:648] HybridClock initialized: now 1714702739497233 us; error 41 us; skew 500 ppm
I20240503 02:18:59.497596 30180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:18:59.499902 30180 webserver.cc:460] Webserver started at http://127.0.145.6:46155/ using document root <none> and password file <none>
I20240503 02:18:59.500630 30180 fs_manager.cc:359] Metadata directory not provided
I20240503 02:18:59.500746 30180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:18:59.501050 30180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:18:59.503880 30180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "58a2474aadf746cc9d50a38f1ee9f80c"
format_stamp: "Formatted at 2024-05-03 02:18:59 on dist-test-slave-gq9h"
I20240503 02:18:59.504688 30180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "58a2474aadf746cc9d50a38f1ee9f80c"
format_stamp: "Formatted at 2024-05-03 02:18:59 on dist-test-slave-gq9h"
I20240503 02:18:59.509997 30180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240503 02:18:59.513525 30194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:59.515004 30180 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240503 02:18:59.515165 30180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "58a2474aadf746cc9d50a38f1ee9f80c"
format_stamp: "Formatted at 2024-05-03 02:18:59 on dist-test-slave-gq9h"
I20240503 02:18:59.515399 30180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:18:59.546893 30180 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:18:59.547804 30180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:18:59.548081 30180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:18:59.549031 30180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:18:59.550716 30180 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:18:59.550817 30180 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:59.550930 30180 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:18:59.551012 30180 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:18:59.582636 30180 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.6:39629
I20240503 02:18:59.582692 30306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.6:39629 every 8 connection(s)
I20240503 02:18:59.583976 30180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:18:59.588742   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 30180
I20240503 02:18:59.588937   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:18:59.597575 30307 heartbeater.cc:347] Connected to a master server at 127.0.145.62:44317
I20240503 02:18:59.597887 30307 heartbeater.cc:464] Registering TS with master...
I20240503 02:18:59.598762 30307 heartbeater.cc:511] Master 127.0.145.62:44317 requested a full tablet report, sending...
I20240503 02:18:59.647455 29465 ts_manager.cc:194] Registered new tserver with Master: 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6:39629)
I20240503 02:18:59.648461 29465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.6:32875
I20240503 02:18:59.658912   580 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 02:18:59.675717   580 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240503 02:18:59.675927   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 30180
I20240503 02:18:59.686381 30302 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:18:59.705307 30045 heartbeater.cc:503] Master 127.0.145.62:44317 was elected leader, sending a full tablet report...
I20240503 02:18:59.769834   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 30180
I20240503 02:18:59.786731   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 29917
I20240503 02:18:59.794574 30040 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:18:59.877034   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 29917
I20240503 02:18:59.893591   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 30048
I20240503 02:18:59.901348 30171 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:18:59.985399   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 30048
I20240503 02:19:03.754380 29465 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39094:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 02:19:03.761472 29465 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:19:03.792088 29849 tablet_service.cc:1467] Processing CreateTablet for tablet 9e13aebe3d3649dda2b2a4f157c08dbc (DEFAULT_TABLE table=rebalance_test_table_0 [id=dab45517a1ea4baa80835e9059e98803]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:03.792186 29584 tablet_service.cc:1467] Processing CreateTablet for tablet 91f62fd5389d4ee7ac179ed61ecd858a (DEFAULT_TABLE table=rebalance_test_table_0 [id=dab45517a1ea4baa80835e9059e98803]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:03.793430 29849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e13aebe3d3649dda2b2a4f157c08dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:03.793572 29584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f62fd5389d4ee7ac179ed61ecd858a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:03.795842 29847 tablet_service.cc:1467] Processing CreateTablet for tablet 91f62fd5389d4ee7ac179ed61ecd858a (DEFAULT_TABLE table=rebalance_test_table_0 [id=dab45517a1ea4baa80835e9059e98803]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:03.796394 29847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f62fd5389d4ee7ac179ed61ecd858a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:03.797569 29848 tablet_service.cc:1467] Processing CreateTablet for tablet 548b1357845a405aad60446d2c5d120c (DEFAULT_TABLE table=rebalance_test_table_0 [id=dab45517a1ea4baa80835e9059e98803]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:03.798077 29848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548b1357845a405aad60446d2c5d120c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:03.798851 29585 tablet_service.cc:1467] Processing CreateTablet for tablet 548b1357845a405aad60446d2c5d120c (DEFAULT_TABLE table=rebalance_test_table_0 [id=dab45517a1ea4baa80835e9059e98803]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:03.799348 29585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548b1357845a405aad60446d2c5d120c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:03.801892 29586 tablet_service.cc:1467] Processing CreateTablet for tablet 9e13aebe3d3649dda2b2a4f157c08dbc (DEFAULT_TABLE table=rebalance_test_table_0 [id=dab45517a1ea4baa80835e9059e98803]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:03.802387 29586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e13aebe3d3649dda2b2a4f157c08dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:03.795202 29717 tablet_service.cc:1467] Processing CreateTablet for tablet 548b1357845a405aad60446d2c5d120c (DEFAULT_TABLE table=rebalance_test_table_0 [id=dab45517a1ea4baa80835e9059e98803]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:03.812762 29717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548b1357845a405aad60446d2c5d120c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:03.813809 30333 tablet_bootstrap.cc:492] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25: Bootstrap starting.
I20240503 02:19:03.794530 29718 tablet_service.cc:1467] Processing CreateTablet for tablet 9e13aebe3d3649dda2b2a4f157c08dbc (DEFAULT_TABLE table=rebalance_test_table_0 [id=dab45517a1ea4baa80835e9059e98803]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:03.795159 29716 tablet_service.cc:1467] Processing CreateTablet for tablet 91f62fd5389d4ee7ac179ed61ecd858a (DEFAULT_TABLE table=rebalance_test_table_0 [id=dab45517a1ea4baa80835e9059e98803]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:03.815244 29718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e13aebe3d3649dda2b2a4f157c08dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:03.816006 29716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f62fd5389d4ee7ac179ed61ecd858a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:03.818960 30333 tablet_bootstrap.cc:654] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:03.820315 30333 log.cc:824] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25: Log is configured to *not* fsync() on all Append() calls
I20240503 02:19:03.830376 30333 tablet_bootstrap.cc:492] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25: No bootstrap required, opened a new log
I20240503 02:19:03.830760 30333 ts_tablet_manager.cc:1367] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent bootstrapping tablet: real 0.017s	user 0.009s	sys 0.004s
I20240503 02:19:03.832864 30336 tablet_bootstrap.cc:492] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: Bootstrap starting.
I20240503 02:19:03.837241 30336 tablet_bootstrap.cc:654] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:03.837700 30334 tablet_bootstrap.cc:492] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc: Bootstrap starting.
I20240503 02:19:03.838532 30336 log.cc:824] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: Log is configured to *not* fsync() on all Append() calls
I20240503 02:19:03.840669 30334 tablet_bootstrap.cc:654] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:03.841547 30336 tablet_bootstrap.cc:492] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: No bootstrap required, opened a new log
I20240503 02:19:03.841333 30333 raft_consensus.cc:348] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } }
I20240503 02:19:03.841660 30334 log.cc:824] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc: Log is configured to *not* fsync() on all Append() calls
I20240503 02:19:03.841843 30336 ts_tablet_manager.cc:1367] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.003s
I20240503 02:19:03.841815 30333 raft_consensus.cc:374] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:03.841948 30333 raft_consensus.cc:729] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c36b0a7f8a3485f9a9d62e3e7793f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:03.842814 30333 consensus_queue.cc:260] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } }
I20240503 02:19:03.844365 30334 tablet_bootstrap.cc:492] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc: No bootstrap required, opened a new log
I20240503 02:19:03.844552 30333 ts_tablet_manager.cc:1398] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent starting tablet: real 0.014s	user 0.011s	sys 0.000s
I20240503 02:19:03.844628 30334 ts_tablet_manager.cc:1367] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240503 02:19:03.844956 30333 tablet_bootstrap.cc:492] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25: Bootstrap starting.
I20240503 02:19:03.846475 30333 tablet_bootstrap.cc:654] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:03.851826 30336 raft_consensus.cc:348] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } }
I20240503 02:19:03.852416 30336 raft_consensus.cc:374] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:03.852551 30336 raft_consensus.cc:729] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f937fbfb7c49f2afe278eaa73edbba, State: Initialized, Role: FOLLOWER
I20240503 02:19:03.852865 30334 raft_consensus.cc:348] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.853267 30334 raft_consensus.cc:374] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:03.853374 30334 raft_consensus.cc:729] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Initialized, Role: FOLLOWER
I20240503 02:19:03.853415 30336 consensus_queue.cc:260] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } }
I20240503 02:19:03.854049 30334 consensus_queue.cc:260] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.854974 30333 tablet_bootstrap.cc:492] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25: No bootstrap required, opened a new log
I20240503 02:19:03.855170 30333 ts_tablet_manager.cc:1367] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.005s
I20240503 02:19:03.856482 30333 raft_consensus.cc:348] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.856750 30333 raft_consensus.cc:374] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:03.856878 30334 ts_tablet_manager.cc:1398] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc: Time spent starting tablet: real 0.012s	user 0.012s	sys 0.000s
I20240503 02:19:03.856854 30333 raft_consensus.cc:729] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c36b0a7f8a3485f9a9d62e3e7793f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:03.857280 30334 tablet_bootstrap.cc:492] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc: Bootstrap starting.
I20240503 02:19:03.857133 30333 consensus_queue.cc:260] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.857939 30333 ts_tablet_manager.cc:1398] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.003s
I20240503 02:19:03.858258 30333 tablet_bootstrap.cc:492] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25: Bootstrap starting.
I20240503 02:19:03.859863 30334 tablet_bootstrap.cc:654] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:03.860225 30333 tablet_bootstrap.cc:654] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:03.862661 30333 tablet_bootstrap.cc:492] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25: No bootstrap required, opened a new log
I20240503 02:19:03.862815 30333 ts_tablet_manager.cc:1367] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:19:03.863230 30336 ts_tablet_manager.cc:1398] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: Time spent starting tablet: real 0.021s	user 0.017s	sys 0.004s
I20240503 02:19:03.863569 30333 raft_consensus.cc:348] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.863687 30336 tablet_bootstrap.cc:492] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba: Bootstrap starting.
I20240503 02:19:03.863794 30333 raft_consensus.cc:374] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:03.863869 30333 raft_consensus.cc:729] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c36b0a7f8a3485f9a9d62e3e7793f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:03.864172 30333 consensus_queue.cc:260] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.864830 30333 ts_tablet_manager.cc:1398] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:19:03.865262 30336 tablet_bootstrap.cc:654] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:03.868501 30334 tablet_bootstrap.cc:492] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc: No bootstrap required, opened a new log
I20240503 02:19:03.868664 30334 ts_tablet_manager.cc:1367] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.005s
I20240503 02:19:03.869441 30334 raft_consensus.cc:348] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.869609 30334 raft_consensus.cc:374] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:03.869695 30334 raft_consensus.cc:729] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Initialized, Role: FOLLOWER
I20240503 02:19:03.869894 30334 consensus_queue.cc:260] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.870781 30334 ts_tablet_manager.cc:1398] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:19:03.871004 30334 tablet_bootstrap.cc:492] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc: Bootstrap starting.
I20240503 02:19:03.871410 30336 tablet_bootstrap.cc:492] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba: No bootstrap required, opened a new log
I20240503 02:19:03.871585 30336 ts_tablet_manager.cc:1367] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.004s
I20240503 02:19:03.872370 30334 tablet_bootstrap.cc:654] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:03.873430 30336 raft_consensus.cc:348] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.873680 30336 raft_consensus.cc:374] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:03.873786 30336 raft_consensus.cc:729] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f937fbfb7c49f2afe278eaa73edbba, State: Initialized, Role: FOLLOWER
I20240503 02:19:03.874019 30336 consensus_queue.cc:260] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.874265 30334 tablet_bootstrap.cc:492] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc: No bootstrap required, opened a new log
I20240503 02:19:03.874424 30334 ts_tablet_manager.cc:1367] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:19:03.874672 30336 ts_tablet_manager.cc:1398] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 02:19:03.874944 30336 tablet_bootstrap.cc:492] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba: Bootstrap starting.
I20240503 02:19:03.875167 30334 raft_consensus.cc:348] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } }
I20240503 02:19:03.875375 30334 raft_consensus.cc:374] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:03.875452 30334 raft_consensus.cc:729] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Initialized, Role: FOLLOWER
I20240503 02:19:03.875694 30334 consensus_queue.cc:260] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } }
I20240503 02:19:03.876315 30334 ts_tablet_manager.cc:1398] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:19:03.876505 30336 tablet_bootstrap.cc:654] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:03.879307 30336 tablet_bootstrap.cc:492] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba: No bootstrap required, opened a new log
I20240503 02:19:03.879462 30336 ts_tablet_manager.cc:1367] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240503 02:19:03.880020 30336 raft_consensus.cc:348] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.880184 30336 raft_consensus.cc:374] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:03.880239 30336 raft_consensus.cc:729] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f937fbfb7c49f2afe278eaa73edbba, State: Initialized, Role: FOLLOWER
I20240503 02:19:03.880434 30336 consensus_queue.cc:260] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.880892 30336 ts_tablet_manager.cc:1398] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:19:03.883857 30341 raft_consensus.cc:482] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:03.884054 30341 raft_consensus.cc:504] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.885594 30341 leader_election.cc:288] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:03.895388 29606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f62fd5389d4ee7ac179ed61ecd858a" candidate_uuid: "99f937fbfb7c49f2afe278eaa73edbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bd09ad30cc4932bfff380344c8e9fc" is_pre_election: true
I20240503 02:19:03.895388 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f62fd5389d4ee7ac179ed61ecd858a" candidate_uuid: "99f937fbfb7c49f2afe278eaa73edbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" is_pre_election: true
I20240503 02:19:03.896003 29869 raft_consensus.cc:2454] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f937fbfb7c49f2afe278eaa73edbba in term 0.
I20240503 02:19:03.896003 29606 raft_consensus.cc:2454] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f937fbfb7c49f2afe278eaa73edbba in term 0.
I20240503 02:19:03.896696 29673 leader_election.cc:302] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c36b0a7f8a3485f9a9d62e3e7793f25, 99f937fbfb7c49f2afe278eaa73edbba; no voters: 
I20240503 02:19:03.897150 30341 raft_consensus.cc:2781] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:03.897289 30341 raft_consensus.cc:482] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:03.897383 30341 raft_consensus.cc:3037] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:03.899463 30341 raft_consensus.cc:504] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.900115 30341 leader_election.cc:288] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [CANDIDATE]: Term 1 election: Requested vote from peers 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:03.900564 29606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f62fd5389d4ee7ac179ed61ecd858a" candidate_uuid: "99f937fbfb7c49f2afe278eaa73edbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bd09ad30cc4932bfff380344c8e9fc"
I20240503 02:19:03.900592 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f62fd5389d4ee7ac179ed61ecd858a" candidate_uuid: "99f937fbfb7c49f2afe278eaa73edbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
I20240503 02:19:03.900748 29606 raft_consensus.cc:3037] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:03.900766 29869 raft_consensus.cc:3037] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:03.902909 29869 raft_consensus.cc:2454] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f937fbfb7c49f2afe278eaa73edbba in term 1.
I20240503 02:19:03.902916 29606 raft_consensus.cc:2454] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f937fbfb7c49f2afe278eaa73edbba in term 1.
I20240503 02:19:03.903334 29672 leader_election.cc:302] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 99f937fbfb7c49f2afe278eaa73edbba; no voters: 
I20240503 02:19:03.903631 30341 raft_consensus.cc:2781] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:03.904047 30341 raft_consensus.cc:686] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [term 1 LEADER]: Becoming Leader. State: Replica: 99f937fbfb7c49f2afe278eaa73edbba, State: Running, Role: LEADER
I20240503 02:19:03.904512 30341 consensus_queue.cc:237] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.909070 29465 catalog_manager.cc:5484] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2). New cstate: current_term: 1 leader_uuid: "99f937fbfb7c49f2afe278eaa73edbba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: UNKNOWN } } }
W20240503 02:19:03.926218 29915 tablet.cc:2369] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:19:03.950866 29784 tablet.cc:2369] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:19:03.961941 30342 raft_consensus.cc:482] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:03.962152 30342 raft_consensus.cc:504] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.964138 30342 leader_election.cc:288] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:03.978447 30341 raft_consensus.cc:482] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:03.978664 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548b1357845a405aad60446d2c5d120c" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" is_pre_election: true
I20240503 02:19:03.978813 30341 raft_consensus.cc:504] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.978967 29869 raft_consensus.cc:2454] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 0.
I20240503 02:19:03.979564 30341 leader_election.cc:288] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:03.979537 29541 leader_election.cc:302] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 6c36b0a7f8a3485f9a9d62e3e7793f25; no voters: 
W20240503 02:19:03.979928 29652 tablet.cc:2369] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:19:03.980281 30342 raft_consensus.cc:2781] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:03.980408 30342 raft_consensus.cc:482] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:03.980507 30342 raft_consensus.cc:3037] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:03.980576 29738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548b1357845a405aad60446d2c5d120c" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba" is_pre_election: true
I20240503 02:19:03.981006 29738 raft_consensus.cc:2454] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 0.
I20240503 02:19:03.981333 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548b1357845a405aad60446d2c5d120c" candidate_uuid: "99f937fbfb7c49f2afe278eaa73edbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" is_pre_election: true
I20240503 02:19:03.981371 29606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548b1357845a405aad60446d2c5d120c" candidate_uuid: "99f937fbfb7c49f2afe278eaa73edbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bd09ad30cc4932bfff380344c8e9fc" is_pre_election: true
I20240503 02:19:03.981564 29869 raft_consensus.cc:2454] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f937fbfb7c49f2afe278eaa73edbba in term 0.
I20240503 02:19:03.981957 29673 leader_election.cc:302] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c36b0a7f8a3485f9a9d62e3e7793f25, 99f937fbfb7c49f2afe278eaa73edbba; no voters: 
I20240503 02:19:03.982259 30341 raft_consensus.cc:2781] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:03.982345 30341 raft_consensus.cc:482] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:03.982443 30341 raft_consensus.cc:3037] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:03.983579 30342 raft_consensus.cc:504] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.984230 29606 raft_consensus.cc:2379] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99f937fbfb7c49f2afe278eaa73edbba in current term 1: Already voted for candidate 07bd09ad30cc4932bfff380344c8e9fc in this term.
I20240503 02:19:03.984509 30342 leader_election.cc:288] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 election: Requested vote from peers 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:03.984828 29738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548b1357845a405aad60446d2c5d120c" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba"
I20240503 02:19:03.985198 30341 raft_consensus.cc:504] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.985303 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548b1357845a405aad60446d2c5d120c" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
I20240503 02:19:03.985496 29869 raft_consensus.cc:3037] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:03.985944 30341 leader_election.cc:288] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [CANDIDATE]: Term 1 election: Requested vote from peers 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:03.986972 29605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548b1357845a405aad60446d2c5d120c" candidate_uuid: "99f937fbfb7c49f2afe278eaa73edbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bd09ad30cc4932bfff380344c8e9fc"
I20240503 02:19:03.986999 29868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "548b1357845a405aad60446d2c5d120c" candidate_uuid: "99f937fbfb7c49f2afe278eaa73edbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
I20240503 02:19:03.987279 29605 raft_consensus.cc:2379] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99f937fbfb7c49f2afe278eaa73edbba in current term 1: Already voted for candidate 07bd09ad30cc4932bfff380344c8e9fc in this term.
I20240503 02:19:03.987974 29673 leader_election.cc:302] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99f937fbfb7c49f2afe278eaa73edbba; no voters: 07bd09ad30cc4932bfff380344c8e9fc, 6c36b0a7f8a3485f9a9d62e3e7793f25
I20240503 02:19:03.988332 29869 raft_consensus.cc:2454] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 1.
I20240503 02:19:03.988464 30341 raft_consensus.cc:2726] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:19:03.988775 29738 raft_consensus.cc:2379] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 07bd09ad30cc4932bfff380344c8e9fc in current term 1: Already voted for candidate 99f937fbfb7c49f2afe278eaa73edbba in this term.
I20240503 02:19:03.988775 29541 leader_election.cc:302] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 6c36b0a7f8a3485f9a9d62e3e7793f25; no voters: 
I20240503 02:19:03.989070 30342 raft_consensus.cc:2781] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:03.989492 30342 raft_consensus.cc:686] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Becoming Leader. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Running, Role: LEADER
I20240503 02:19:03.990031 30342 consensus_queue.cc:237] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:03.994762 29465 catalog_manager.cc:5484] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1). New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:04.104820 30342 raft_consensus.cc:482] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:04.105108 30342 raft_consensus.cc:504] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } }
I20240503 02:19:04.105720 30342 leader_election.cc:288] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:04.106240 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e13aebe3d3649dda2b2a4f157c08dbc" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" is_pre_election: true
I20240503 02:19:04.106266 29738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e13aebe3d3649dda2b2a4f157c08dbc" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba" is_pre_election: true
I20240503 02:19:04.106470 29869 raft_consensus.cc:2454] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 0.
I20240503 02:19:04.106492 29738 raft_consensus.cc:2454] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 0.
I20240503 02:19:04.106920 29541 leader_election.cc:302] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 6c36b0a7f8a3485f9a9d62e3e7793f25; no voters: 
I20240503 02:19:04.107221 30342 raft_consensus.cc:2781] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:04.107306 30342 raft_consensus.cc:482] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:04.107416 30342 raft_consensus.cc:3037] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:04.109318 30342 raft_consensus.cc:504] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } }
I20240503 02:19:04.109935 30342 leader_election.cc:288] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 election: Requested vote from peers 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:04.110359 29738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e13aebe3d3649dda2b2a4f157c08dbc" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba"
I20240503 02:19:04.110466 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e13aebe3d3649dda2b2a4f157c08dbc" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
I20240503 02:19:04.110553 29738 raft_consensus.cc:3037] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:04.110699 29869 raft_consensus.cc:3037] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:04.112509 29738 raft_consensus.cc:2454] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 1.
I20240503 02:19:04.112504 29869 raft_consensus.cc:2454] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 1.
I20240503 02:19:04.112906 29541 leader_election.cc:302] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 6c36b0a7f8a3485f9a9d62e3e7793f25; no voters: 
I20240503 02:19:04.113207 30342 raft_consensus.cc:2781] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:04.113373 30342 raft_consensus.cc:686] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Becoming Leader. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Running, Role: LEADER
I20240503 02:19:04.113642 30342 consensus_queue.cc:237] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } }
I20240503 02:19:04.116202 29465 catalog_manager.cc:5484] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1). New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } }
I20240503 02:19:04.296818 30341 consensus_queue.cc:1035] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:04.303825 30361 consensus_queue.cc:1035] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:04.376935 30342 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:04.382755 30342 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:19:04.520390 30357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:04.520769 30357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:04.528254 30357 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 02:19:04.566349 30342 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:04.572182 30342 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:19:04.860986 29539 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.2:37371, user_credentials={real_user=slave}} blocked reactor thread for 111617us

INSERT report
    rows total: 2000
    time total: 301.176 ms
  time per row: 0.150588 ms
W20240503 02:19:04.886865 29541 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.145.3:41133, user_credentials={real_user=slave}} blocked reactor thread for 135844us
-----------------------------------------------------
Suppressions used:
  count      bytes template
     14       3624 libcrypto.so.1.1
-----------------------------------------------------

I20240503 02:19:05.899756 29464 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39094:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 02:19:05.900425 29464 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:19:05.908403 29585 tablet_service.cc:1467] Processing CreateTablet for tablet 2451cf2c781a44e0882f111cb29fa40f (DEFAULT_TABLE table=rebalance_test_table_1 [id=11a9093ed71046c1b45b6520ebcc4ed0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:05.908519 29716 tablet_service.cc:1467] Processing CreateTablet for tablet 2451cf2c781a44e0882f111cb29fa40f (DEFAULT_TABLE table=rebalance_test_table_1 [id=11a9093ed71046c1b45b6520ebcc4ed0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:05.908915 29585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2451cf2c781a44e0882f111cb29fa40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:05.908965 29716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2451cf2c781a44e0882f111cb29fa40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:05.909240 29718 tablet_service.cc:1467] Processing CreateTablet for tablet 1b22e23ebdee48358f75d140d3681800 (DEFAULT_TABLE table=rebalance_test_table_1 [id=11a9093ed71046c1b45b6520ebcc4ed0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:05.909600 29718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b22e23ebdee48358f75d140d3681800. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:05.911152 29849 tablet_service.cc:1467] Processing CreateTablet for tablet 5e14a9b5731446669effe472c442cda1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=11a9093ed71046c1b45b6520ebcc4ed0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:05.911530 29849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e14a9b5731446669effe472c442cda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:05.915086 29848 tablet_service.cc:1467] Processing CreateTablet for tablet 2451cf2c781a44e0882f111cb29fa40f (DEFAULT_TABLE table=rebalance_test_table_1 [id=11a9093ed71046c1b45b6520ebcc4ed0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:05.915244 29717 tablet_service.cc:1467] Processing CreateTablet for tablet 5e14a9b5731446669effe472c442cda1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=11a9093ed71046c1b45b6520ebcc4ed0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:05.916059 29717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e14a9b5731446669effe472c442cda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:05.916047 29848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2451cf2c781a44e0882f111cb29fa40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:05.919361 29847 tablet_service.cc:1467] Processing CreateTablet for tablet 1b22e23ebdee48358f75d140d3681800 (DEFAULT_TABLE table=rebalance_test_table_1 [id=11a9093ed71046c1b45b6520ebcc4ed0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:05.919744 29847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b22e23ebdee48358f75d140d3681800. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:05.921355 29586 tablet_service.cc:1467] Processing CreateTablet for tablet 1b22e23ebdee48358f75d140d3681800 (DEFAULT_TABLE table=rebalance_test_table_1 [id=11a9093ed71046c1b45b6520ebcc4ed0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:05.921753 29586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b22e23ebdee48358f75d140d3681800. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:05.924871 29584 tablet_service.cc:1467] Processing CreateTablet for tablet 5e14a9b5731446669effe472c442cda1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=11a9093ed71046c1b45b6520ebcc4ed0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:05.925256 29584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e14a9b5731446669effe472c442cda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:05.929246 30424 tablet_bootstrap.cc:492] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Bootstrap starting.
I20240503 02:19:05.931047 30424 tablet_bootstrap.cc:654] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:05.933211 30425 tablet_bootstrap.cc:492] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc: Bootstrap starting.
I20240503 02:19:05.935604 30425 tablet_bootstrap.cc:654] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:05.940253 30423 tablet_bootstrap.cc:492] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba: Bootstrap starting.
I20240503 02:19:05.941900 30424 tablet_bootstrap.cc:492] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25: No bootstrap required, opened a new log
I20240503 02:19:05.941957 30423 tablet_bootstrap.cc:654] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:05.942132 30424 ts_tablet_manager.cc:1367] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent bootstrapping tablet: real 0.013s	user 0.008s	sys 0.004s
I20240503 02:19:05.942167 30425 tablet_bootstrap.cc:492] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc: No bootstrap required, opened a new log
I20240503 02:19:05.942288 30425 ts_tablet_manager.cc:1367] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.004s
I20240503 02:19:05.943012 30424 raft_consensus.cc:348] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.942986 30425 raft_consensus.cc:348] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.943186 30424 raft_consensus.cc:374] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:05.943199 30425 raft_consensus.cc:374] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:05.943289 30424 raft_consensus.cc:729] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c36b0a7f8a3485f9a9d62e3e7793f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:05.943303 30425 raft_consensus.cc:729] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Initialized, Role: FOLLOWER
I20240503 02:19:05.943554 30424 consensus_queue.cc:260] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.943554 30425 consensus_queue.cc:260] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.944231 30425 ts_tablet_manager.cc:1398] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:19:05.944548 30425 tablet_bootstrap.cc:492] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc: Bootstrap starting.
I20240503 02:19:05.944864 30423 tablet_bootstrap.cc:492] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba: No bootstrap required, opened a new log
I20240503 02:19:05.945061 30423 ts_tablet_manager.cc:1367] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 02:19:05.946017 30423 raft_consensus.cc:348] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.946327 30425 tablet_bootstrap.cc:654] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:05.947093 30423 raft_consensus.cc:374] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:05.947240 30423 raft_consensus.cc:729] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f937fbfb7c49f2afe278eaa73edbba, State: Initialized, Role: FOLLOWER
I20240503 02:19:05.947567 30423 consensus_queue.cc:260] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.949836 30424 ts_tablet_manager.cc:1398] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20240503 02:19:05.950078 30425 tablet_bootstrap.cc:492] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc: No bootstrap required, opened a new log
I20240503 02:19:05.950174 30424 tablet_bootstrap.cc:492] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Bootstrap starting.
I20240503 02:19:05.950235 30425 ts_tablet_manager.cc:1367] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240503 02:19:05.951102 30423 ts_tablet_manager.cc:1398] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240503 02:19:05.950994 30425 raft_consensus.cc:348] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.951223 30425 raft_consensus.cc:374] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:05.951308 30425 raft_consensus.cc:729] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Initialized, Role: FOLLOWER
I20240503 02:19:05.951421 30423 tablet_bootstrap.cc:492] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba: Bootstrap starting.
I20240503 02:19:05.951581 30425 consensus_queue.cc:260] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.952193 30425 ts_tablet_manager.cc:1398] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:19:05.952481 30425 tablet_bootstrap.cc:492] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc: Bootstrap starting.
I20240503 02:19:05.954041 30423 tablet_bootstrap.cc:654] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:05.954463 30425 tablet_bootstrap.cc:654] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:05.955850 30424 tablet_bootstrap.cc:654] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:05.957218 30425 tablet_bootstrap.cc:492] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc: No bootstrap required, opened a new log
I20240503 02:19:05.957376 30425 ts_tablet_manager.cc:1367] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 02:19:05.957983 30424 tablet_bootstrap.cc:492] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25: No bootstrap required, opened a new log
I20240503 02:19:05.957924 30425 raft_consensus.cc:348] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.958076 30425 raft_consensus.cc:374] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:05.958160 30425 raft_consensus.cc:729] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Initialized, Role: FOLLOWER
I20240503 02:19:05.958191 30424 ts_tablet_manager.cc:1367] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.000s
I20240503 02:19:05.958369 30425 consensus_queue.cc:260] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.958827 30425 ts_tablet_manager.cc:1398] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240503 02:19:05.958997 30424 raft_consensus.cc:348] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.959220 30424 raft_consensus.cc:374] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:05.959272 30424 raft_consensus.cc:729] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c36b0a7f8a3485f9a9d62e3e7793f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:05.959461 30424 consensus_queue.cc:260] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.959951 30423 tablet_bootstrap.cc:492] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba: No bootstrap required, opened a new log
I20240503 02:19:05.960094 30423 ts_tablet_manager.cc:1367] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I20240503 02:19:05.961416 30423 raft_consensus.cc:348] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.961549 30424 ts_tablet_manager.cc:1398] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:19:05.961663 30423 raft_consensus.cc:374] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:05.961755 30423 raft_consensus.cc:729] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f937fbfb7c49f2afe278eaa73edbba, State: Initialized, Role: FOLLOWER
I20240503 02:19:05.962008 30424 tablet_bootstrap.cc:492] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25: Bootstrap starting.
I20240503 02:19:05.962033 30423 consensus_queue.cc:260] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.963526 30424 tablet_bootstrap.cc:654] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:05.965873 30424 tablet_bootstrap.cc:492] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25: No bootstrap required, opened a new log
I20240503 02:19:05.965994 30424 ts_tablet_manager.cc:1367] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:19:05.966641 30424 raft_consensus.cc:348] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.966809 30424 raft_consensus.cc:374] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:05.966878 30424 raft_consensus.cc:729] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c36b0a7f8a3485f9a9d62e3e7793f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:05.967106 30424 consensus_queue.cc:260] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.967536 30424 ts_tablet_manager.cc:1398] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.002s
I20240503 02:19:05.972081 30423 ts_tablet_manager.cc:1398] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.000s
I20240503 02:19:05.972932 30423 tablet_bootstrap.cc:492] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba: Bootstrap starting.
I20240503 02:19:05.974750 30423 tablet_bootstrap.cc:654] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:05.976773 30423 tablet_bootstrap.cc:492] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba: No bootstrap required, opened a new log
I20240503 02:19:05.976909 30423 ts_tablet_manager.cc:1367] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 02:19:05.977638 30423 raft_consensus.cc:348] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.977813 30423 raft_consensus.cc:374] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:05.977897 30423 raft_consensus.cc:729] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f937fbfb7c49f2afe278eaa73edbba, State: Initialized, Role: FOLLOWER
I20240503 02:19:05.978107 30423 consensus_queue.cc:260] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:05.978658 30423 ts_tablet_manager.cc:1398] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:19:06.070112 30361 raft_consensus.cc:482] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:06.070361 30361 raft_consensus.cc:504] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:06.071080 30361 leader_election.cc:288] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:06.071633 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e14a9b5731446669effe472c442cda1" candidate_uuid: "99f937fbfb7c49f2afe278eaa73edbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" is_pre_election: true
I20240503 02:19:06.071638 29606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e14a9b5731446669effe472c442cda1" candidate_uuid: "99f937fbfb7c49f2afe278eaa73edbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bd09ad30cc4932bfff380344c8e9fc" is_pre_election: true
I20240503 02:19:06.071861 29869 raft_consensus.cc:2454] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f937fbfb7c49f2afe278eaa73edbba in term 0.
I20240503 02:19:06.071863 29606 raft_consensus.cc:2454] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f937fbfb7c49f2afe278eaa73edbba in term 0.
I20240503 02:19:06.072307 29672 leader_election.cc:302] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 99f937fbfb7c49f2afe278eaa73edbba; no voters: 
I20240503 02:19:06.072619 30361 raft_consensus.cc:2781] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:06.072719 30361 raft_consensus.cc:482] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:06.072826 30361 raft_consensus.cc:3037] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:06.074817 30361 raft_consensus.cc:504] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:06.075592 30361 leader_election.cc:288] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [CANDIDATE]: Term 1 election: Requested vote from peers 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:06.075871 29606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e14a9b5731446669effe472c442cda1" candidate_uuid: "99f937fbfb7c49f2afe278eaa73edbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bd09ad30cc4932bfff380344c8e9fc"
I20240503 02:19:06.075906 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e14a9b5731446669effe472c442cda1" candidate_uuid: "99f937fbfb7c49f2afe278eaa73edbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
I20240503 02:19:06.076082 29606 raft_consensus.cc:3037] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:06.076108 29869 raft_consensus.cc:3037] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:06.078006 29606 raft_consensus.cc:2454] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f937fbfb7c49f2afe278eaa73edbba in term 1.
I20240503 02:19:06.078143 29869 raft_consensus.cc:2454] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f937fbfb7c49f2afe278eaa73edbba in term 1.
I20240503 02:19:06.078487 29672 leader_election.cc:302] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 99f937fbfb7c49f2afe278eaa73edbba; no voters: 
I20240503 02:19:06.078794 30361 raft_consensus.cc:2781] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:06.079169 30361 raft_consensus.cc:686] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 LEADER]: Becoming Leader. State: Replica: 99f937fbfb7c49f2afe278eaa73edbba, State: Running, Role: LEADER
I20240503 02:19:06.079402 30361 consensus_queue.cc:237] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:06.081979 29463 catalog_manager.cc:5484] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2). New cstate: current_term: 1 leader_uuid: "99f937fbfb7c49f2afe278eaa73edbba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:06.218133 30382 raft_consensus.cc:482] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:06.218294 30382 raft_consensus.cc:504] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:06.218971 30382 leader_election.cc:288] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:06.219354 29738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2451cf2c781a44e0882f111cb29fa40f" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba" is_pre_election: true
I20240503 02:19:06.219502 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2451cf2c781a44e0882f111cb29fa40f" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" is_pre_election: true
I20240503 02:19:06.219621 29738 raft_consensus.cc:2454] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 0.
I20240503 02:19:06.219750 29869 raft_consensus.cc:2454] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 0.
I20240503 02:19:06.220100 29539 leader_election.cc:302] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 99f937fbfb7c49f2afe278eaa73edbba; no voters: 
I20240503 02:19:06.220367 30382 raft_consensus.cc:2781] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:06.220480 30382 raft_consensus.cc:482] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:06.220583 30382 raft_consensus.cc:3037] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:06.223129 30382 raft_consensus.cc:504] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:06.224082 30382 leader_election.cc:288] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 election: Requested vote from peers 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:06.224185 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2451cf2c781a44e0882f111cb29fa40f" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
I20240503 02:19:06.224164 29738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2451cf2c781a44e0882f111cb29fa40f" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba"
I20240503 02:19:06.224357 29869 raft_consensus.cc:3037] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:06.224375 29738 raft_consensus.cc:3037] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:06.227130 29738 raft_consensus.cc:2454] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 1.
I20240503 02:19:06.227151 29869 raft_consensus.cc:2454] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 1.
I20240503 02:19:06.227595 29541 leader_election.cc:302] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 6c36b0a7f8a3485f9a9d62e3e7793f25; no voters: 
I20240503 02:19:06.227929 30382 raft_consensus.cc:2781] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:06.228304 30382 raft_consensus.cc:686] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Becoming Leader. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Running, Role: LEADER
I20240503 02:19:06.228593 30382 consensus_queue.cc:237] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:06.231480 29463 catalog_manager.cc:5484] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1). New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:06.262308 30382 raft_consensus.cc:482] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:06.262538 30382 raft_consensus.cc:504] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:06.263240 30382 leader_election.cc:288] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:06.263761 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b22e23ebdee48358f75d140d3681800" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" is_pre_election: true
I20240503 02:19:06.263787 29738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b22e23ebdee48358f75d140d3681800" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba" is_pre_election: true
I20240503 02:19:06.263994 29869 raft_consensus.cc:2454] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 0.
I20240503 02:19:06.264010 29738 raft_consensus.cc:2454] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 0.
I20240503 02:19:06.264427 29541 leader_election.cc:302] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 6c36b0a7f8a3485f9a9d62e3e7793f25; no voters: 
I20240503 02:19:06.264695 30382 raft_consensus.cc:2781] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:06.264789 30382 raft_consensus.cc:482] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:06.264904 30382 raft_consensus.cc:3037] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:06.266733 30382 raft_consensus.cc:504] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:06.267297 30382 leader_election.cc:288] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 election: Requested vote from peers 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:06.267737 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b22e23ebdee48358f75d140d3681800" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
I20240503 02:19:06.267751 29738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b22e23ebdee48358f75d140d3681800" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba"
I20240503 02:19:06.267925 29869 raft_consensus.cc:3037] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:06.267932 29738 raft_consensus.cc:3037] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:06.269665 29869 raft_consensus.cc:2454] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 1.
I20240503 02:19:06.269698 29738 raft_consensus.cc:2454] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 1.
I20240503 02:19:06.270058 29541 leader_election.cc:302] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 6c36b0a7f8a3485f9a9d62e3e7793f25; no voters: 
I20240503 02:19:06.270299 30382 raft_consensus.cc:2781] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:06.270457 30382 raft_consensus.cc:686] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Becoming Leader. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Running, Role: LEADER
I20240503 02:19:06.270751 30382 consensus_queue.cc:237] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:06.273213 29463 catalog_manager.cc:5484] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1). New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:06.540751 30361 consensus_queue.cc:1035] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:06.547711 30436 consensus_queue.cc:1035] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:19:06.627611 30441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:06.627915 30441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:06.635354 30441 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:19:06.648263 30382 consensus_queue.cc:1035] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:06.652915 30438 consensus_queue.cc:1035] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240503 02:19:06.787345 30438 consensus_queue.cc:1035] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:19:06.796193 30438 consensus_queue.cc:1035] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 206.368 ms
  time per row: 0.103184 ms
I20240503 02:19:06.991523 29464 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39094:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 02:19:06.992198 29464 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:19:07.000180 29586 tablet_service.cc:1467] Processing CreateTablet for tablet f83faf506c3e49d4ade5f48881d9d1a7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a25280dd90f64d31a487b2c771383e8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:07.000264 29718 tablet_service.cc:1467] Processing CreateTablet for tablet f83faf506c3e49d4ade5f48881d9d1a7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a25280dd90f64d31a487b2c771383e8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:07.000675 29586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83faf506c3e49d4ade5f48881d9d1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:07.000669 29718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83faf506c3e49d4ade5f48881d9d1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:07.000952 29585 tablet_service.cc:1467] Processing CreateTablet for tablet 899b3fc0c13540e485d0b76417c205e4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a25280dd90f64d31a487b2c771383e8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:07.001339 29585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899b3fc0c13540e485d0b76417c205e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:07.003878 29717 tablet_service.cc:1467] Processing CreateTablet for tablet 899b3fc0c13540e485d0b76417c205e4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a25280dd90f64d31a487b2c771383e8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:07.004112 29584 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4fdecce7bd47b9a92d66a775211292 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a25280dd90f64d31a487b2c771383e8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:07.004156 29716 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4fdecce7bd47b9a92d66a775211292 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a25280dd90f64d31a487b2c771383e8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:07.004321 29717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899b3fc0c13540e485d0b76417c205e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:07.004513 29584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4fdecce7bd47b9a92d66a775211292. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:07.004513 29716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4fdecce7bd47b9a92d66a775211292. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:07.007599 29848 tablet_service.cc:1467] Processing CreateTablet for tablet f83faf506c3e49d4ade5f48881d9d1a7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a25280dd90f64d31a487b2c771383e8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:07.007723 29849 tablet_service.cc:1467] Processing CreateTablet for tablet 899b3fc0c13540e485d0b76417c205e4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a25280dd90f64d31a487b2c771383e8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:07.008055 29848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83faf506c3e49d4ade5f48881d9d1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:07.008203 29847 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4fdecce7bd47b9a92d66a775211292 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a25280dd90f64d31a487b2c771383e8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:07.014958 29847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4fdecce7bd47b9a92d66a775211292. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:07.015568 30498 tablet_bootstrap.cc:492] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc: Bootstrap starting.
I20240503 02:19:07.017879 30498 tablet_bootstrap.cc:654] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:07.022758 29849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899b3fc0c13540e485d0b76417c205e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:07.026892 30501 tablet_bootstrap.cc:492] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Bootstrap starting.
I20240503 02:19:07.028080 30498 tablet_bootstrap.cc:492] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc: No bootstrap required, opened a new log
I20240503 02:19:07.028304 30498 ts_tablet_manager.cc:1367] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc: Time spent bootstrapping tablet: real 0.013s	user 0.000s	sys 0.011s
I20240503 02:19:07.029114 30501 tablet_bootstrap.cc:654] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:07.029358 30498 raft_consensus.cc:348] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.029707 30498 raft_consensus.cc:374] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:07.029776 30499 tablet_bootstrap.cc:492] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba: Bootstrap starting.
I20240503 02:19:07.029857 30498 raft_consensus.cc:729] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Initialized, Role: FOLLOWER
I20240503 02:19:07.030156 30498 consensus_queue.cc:260] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.030980 30498 ts_tablet_manager.cc:1398] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:19:07.031419 30498 tablet_bootstrap.cc:492] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc: Bootstrap starting.
I20240503 02:19:07.033632 30498 tablet_bootstrap.cc:654] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:07.034176 30499 tablet_bootstrap.cc:654] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:07.037516 30499 tablet_bootstrap.cc:492] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba: No bootstrap required, opened a new log
I20240503 02:19:07.037539 30501 tablet_bootstrap.cc:492] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: No bootstrap required, opened a new log
I20240503 02:19:07.037695 30499 ts_tablet_manager.cc:1367] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20240503 02:19:07.037691 30501 ts_tablet_manager.cc:1367] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent bootstrapping tablet: real 0.011s	user 0.007s	sys 0.003s
I20240503 02:19:07.038244 30498 tablet_bootstrap.cc:492] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc: No bootstrap required, opened a new log
I20240503 02:19:07.038412 30498 ts_tablet_manager.cc:1367] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240503 02:19:07.038367 30499 raft_consensus.cc:348] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.038579 30499 raft_consensus.cc:374] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:07.038491 30501 raft_consensus.cc:348] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.038714 30499 raft_consensus.cc:729] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f937fbfb7c49f2afe278eaa73edbba, State: Initialized, Role: FOLLOWER
I20240503 02:19:07.038735 30501 raft_consensus.cc:374] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:07.038816 30501 raft_consensus.cc:729] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c36b0a7f8a3485f9a9d62e3e7793f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:07.038962 30499 consensus_queue.cc:260] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.039067 30501 consensus_queue.cc:260] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.039600 30499 ts_tablet_manager.cc:1398] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:19:07.039750 30501 ts_tablet_manager.cc:1398] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:19:07.039880 30499 tablet_bootstrap.cc:492] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba: Bootstrap starting.
I20240503 02:19:07.039222 30498 raft_consensus.cc:348] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.040011 30501 tablet_bootstrap.cc:492] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Bootstrap starting.
I20240503 02:19:07.040041 30498 raft_consensus.cc:374] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:07.040124 30498 raft_consensus.cc:729] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Initialized, Role: FOLLOWER
I20240503 02:19:07.040378 30498 consensus_queue.cc:260] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.041061 30498 ts_tablet_manager.cc:1398] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:19:07.041589 30501 tablet_bootstrap.cc:654] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:07.042084 30499 tablet_bootstrap.cc:654] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:07.042912 30498 tablet_bootstrap.cc:492] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc: Bootstrap starting.
I20240503 02:19:07.045987 30498 tablet_bootstrap.cc:654] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:07.050776 30498 tablet_bootstrap.cc:492] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc: No bootstrap required, opened a new log
I20240503 02:19:07.050760 30501 tablet_bootstrap.cc:492] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25: No bootstrap required, opened a new log
I20240503 02:19:07.050930 30498 ts_tablet_manager.cc:1367] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.002s
I20240503 02:19:07.050951 30501 ts_tablet_manager.cc:1367] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.004s
I20240503 02:19:07.051133 30499 tablet_bootstrap.cc:492] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba: No bootstrap required, opened a new log
I20240503 02:19:07.051301 30499 ts_tablet_manager.cc:1367] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.004s
I20240503 02:19:07.051748 30498 raft_consensus.cc:348] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.051730 30501 raft_consensus.cc:348] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.051954 30498 raft_consensus.cc:374] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:07.051975 30501 raft_consensus.cc:374] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:07.051905 30499 raft_consensus.cc:348] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.052071 30498 raft_consensus.cc:729] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Initialized, Role: FOLLOWER
I20240503 02:19:07.052068 30501 raft_consensus.cc:729] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c36b0a7f8a3485f9a9d62e3e7793f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:07.052152 30499 raft_consensus.cc:374] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:07.052234 30499 raft_consensus.cc:729] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f937fbfb7c49f2afe278eaa73edbba, State: Initialized, Role: FOLLOWER
I20240503 02:19:07.052317 30501 consensus_queue.cc:260] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.052312 30498 consensus_queue.cc:260] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.052495 30499 consensus_queue.cc:260] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.052892 30501 ts_tablet_manager.cc:1398] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:19:07.053067 30498 ts_tablet_manager.cc:1398] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:19:07.053228 30501 tablet_bootstrap.cc:492] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Bootstrap starting.
I20240503 02:19:07.054814 30499 ts_tablet_manager.cc:1398] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:19:07.054996 30501 tablet_bootstrap.cc:654] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:07.055130 30499 tablet_bootstrap.cc:492] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba: Bootstrap starting.
I20240503 02:19:07.057528 30499 tablet_bootstrap.cc:654] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:07.060765 30499 tablet_bootstrap.cc:492] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba: No bootstrap required, opened a new log
I20240503 02:19:07.060907 30499 ts_tablet_manager.cc:1367] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.003s
I20240503 02:19:07.061497 30499 raft_consensus.cc:348] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.061672 30499 raft_consensus.cc:374] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:07.061762 30499 raft_consensus.cc:729] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f937fbfb7c49f2afe278eaa73edbba, State: Initialized, Role: FOLLOWER
I20240503 02:19:07.061972 30499 consensus_queue.cc:260] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.062392 30499 ts_tablet_manager.cc:1398] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:19:07.063704 30501 tablet_bootstrap.cc:492] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25: No bootstrap required, opened a new log
I20240503 02:19:07.063886 30501 ts_tablet_manager.cc:1367] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent bootstrapping tablet: real 0.011s	user 0.010s	sys 0.000s
I20240503 02:19:07.064589 30501 raft_consensus.cc:348] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.064749 30501 raft_consensus.cc:374] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:07.064841 30501 raft_consensus.cc:729] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c36b0a7f8a3485f9a9d62e3e7793f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:07.065066 30501 consensus_queue.cc:260] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.065544 30501 ts_tablet_manager.cc:1398] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:19:07.094853 30489 raft_consensus.cc:482] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:07.095059 30489 raft_consensus.cc:504] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.095757 30489 leader_election.cc:288] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:07.096279 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4fdecce7bd47b9a92d66a775211292" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" is_pre_election: true
I20240503 02:19:07.096287 29736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4fdecce7bd47b9a92d66a775211292" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba" is_pre_election: true
I20240503 02:19:07.096490 29869 raft_consensus.cc:2454] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 0.
I20240503 02:19:07.096501 29736 raft_consensus.cc:2454] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 0.
I20240503 02:19:07.096927 29539 leader_election.cc:302] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 99f937fbfb7c49f2afe278eaa73edbba; no voters: 
I20240503 02:19:07.097208 30489 raft_consensus.cc:2781] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:07.097326 30489 raft_consensus.cc:482] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:07.097417 30489 raft_consensus.cc:3037] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:07.099440 30489 raft_consensus.cc:504] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.100020 30489 leader_election.cc:288] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 election: Requested vote from peers 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:07.100497 29736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4fdecce7bd47b9a92d66a775211292" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba"
I20240503 02:19:07.100536 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4fdecce7bd47b9a92d66a775211292" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
I20240503 02:19:07.100735 29736 raft_consensus.cc:3037] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:07.100752 29869 raft_consensus.cc:3037] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:07.103721 30442 raft_consensus.cc:482] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:07.103718 29736 raft_consensus.cc:2454] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 1.
I20240503 02:19:07.103756 29869 raft_consensus.cc:2454] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 1.
I20240503 02:19:07.103883 30442 raft_consensus.cc:504] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.104203 29541 leader_election.cc:302] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 6c36b0a7f8a3485f9a9d62e3e7793f25; no voters: 
I20240503 02:19:07.104519 30489 raft_consensus.cc:2781] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:07.104676 30489 raft_consensus.cc:686] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Becoming Leader. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Running, Role: LEADER
I20240503 02:19:07.104966 30489 consensus_queue.cc:237] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.105916 30442 leader_election.cc:288] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557), 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371)
I20240503 02:19:07.108943 29459 catalog_manager.cc:5484] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1). New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:07.114078 29736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83faf506c3e49d4ade5f48881d9d1a7" candidate_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba" is_pre_election: true
I20240503 02:19:07.114370 29736 raft_consensus.cc:2454] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c36b0a7f8a3485f9a9d62e3e7793f25 in term 0.
I20240503 02:19:07.114871 29802 leader_election.cc:302] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c36b0a7f8a3485f9a9d62e3e7793f25, 99f937fbfb7c49f2afe278eaa73edbba; no voters: 
I20240503 02:19:07.114849 29606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83faf506c3e49d4ade5f48881d9d1a7" candidate_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bd09ad30cc4932bfff380344c8e9fc" is_pre_election: true
I20240503 02:19:07.115069 29606 raft_consensus.cc:2454] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c36b0a7f8a3485f9a9d62e3e7793f25 in term 0.
I20240503 02:19:07.115240 30442 raft_consensus.cc:2781] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:07.115334 30442 raft_consensus.cc:482] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:07.115396 30442 raft_consensus.cc:3037] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:07.117317 30442 raft_consensus.cc:504] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.117842 30442 leader_election.cc:288] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [CANDIDATE]: Term 1 election: Requested vote from peers 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557), 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371)
I20240503 02:19:07.118299 29606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83faf506c3e49d4ade5f48881d9d1a7" candidate_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bd09ad30cc4932bfff380344c8e9fc"
I20240503 02:19:07.118505 29606 raft_consensus.cc:3037] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:07.118530 29736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83faf506c3e49d4ade5f48881d9d1a7" candidate_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba"
I20240503 02:19:07.118742 29736 raft_consensus.cc:3037] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:07.120491 29606 raft_consensus.cc:2454] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c36b0a7f8a3485f9a9d62e3e7793f25 in term 1.
I20240503 02:19:07.120724 29736 raft_consensus.cc:2454] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c36b0a7f8a3485f9a9d62e3e7793f25 in term 1.
I20240503 02:19:07.120905 29803 leader_election.cc:302] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 6c36b0a7f8a3485f9a9d62e3e7793f25; no voters: 
I20240503 02:19:07.121183 30442 raft_consensus.cc:2781] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:07.121582 30442 raft_consensus.cc:686] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEADER]: Becoming Leader. State: Replica: 6c36b0a7f8a3485f9a9d62e3e7793f25, State: Running, Role: LEADER
I20240503 02:19:07.122071 30442 consensus_queue.cc:237] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.125375 29459 catalog_manager.cc:5484] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3). New cstate: current_term: 1 leader_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: HEALTHY } } }
I20240503 02:19:07.297513 30442 raft_consensus.cc:482] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:07.297768 30442 raft_consensus.cc:504] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.298355 30442 leader_election.cc:288] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557), 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371)
I20240503 02:19:07.298936 29606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899b3fc0c13540e485d0b76417c205e4" candidate_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bd09ad30cc4932bfff380344c8e9fc" is_pre_election: true
I20240503 02:19:07.298955 29736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899b3fc0c13540e485d0b76417c205e4" candidate_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba" is_pre_election: true
I20240503 02:19:07.299177 29736 raft_consensus.cc:2454] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c36b0a7f8a3485f9a9d62e3e7793f25 in term 0.
I20240503 02:19:07.299198 29606 raft_consensus.cc:2454] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c36b0a7f8a3485f9a9d62e3e7793f25 in term 0.
I20240503 02:19:07.299574 29802 leader_election.cc:302] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c36b0a7f8a3485f9a9d62e3e7793f25, 99f937fbfb7c49f2afe278eaa73edbba; no voters: 
I20240503 02:19:07.299839 30442 raft_consensus.cc:2781] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:07.299925 30442 raft_consensus.cc:482] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:07.300051 30442 raft_consensus.cc:3037] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:07.301893 30442 raft_consensus.cc:504] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.302423 30442 leader_election.cc:288] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [CANDIDATE]: Term 1 election: Requested vote from peers 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557), 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371)
I20240503 02:19:07.302919 29736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899b3fc0c13540e485d0b76417c205e4" candidate_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba"
I20240503 02:19:07.302904 29606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899b3fc0c13540e485d0b76417c205e4" candidate_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bd09ad30cc4932bfff380344c8e9fc"
I20240503 02:19:07.303112 29736 raft_consensus.cc:3037] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:07.303120 29606 raft_consensus.cc:3037] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:07.304952 29606 raft_consensus.cc:2454] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c36b0a7f8a3485f9a9d62e3e7793f25 in term 1.
I20240503 02:19:07.304963 29736 raft_consensus.cc:2454] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c36b0a7f8a3485f9a9d62e3e7793f25 in term 1.
I20240503 02:19:07.305320 29802 leader_election.cc:302] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c36b0a7f8a3485f9a9d62e3e7793f25, 99f937fbfb7c49f2afe278eaa73edbba; no voters: 
I20240503 02:19:07.305584 30442 raft_consensus.cc:2781] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:07.305774 30442 raft_consensus.cc:686] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEADER]: Becoming Leader. State: Replica: 6c36b0a7f8a3485f9a9d62e3e7793f25, State: Running, Role: LEADER
I20240503 02:19:07.306041 30442 consensus_queue.cc:237] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } }
I20240503 02:19:07.308703 29460 catalog_manager.cc:5484] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3). New cstate: current_term: 1 leader_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: HEALTHY } } }
I20240503 02:19:07.486750 30489 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:07.493544 30520 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:07.566923 30442 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:07.582227 30513 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:19:07.664129 30516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:07.664458 30516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:07.672358 30516 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 02:19:07.739137 30513 consensus_queue.cc:1035] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:07.753495 30513 consensus_queue.cc:1035] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 280.09 ms
  time per row: 0.140045 ms
I20240503 02:19:08.099165   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.6:39629
--local_ip_for_outbound_sockets=127.0.145.6
--tserver_master_addrs=127.0.145.62:44317
--webserver_port=46155
--webserver_interface=127.0.145.6
--builtin_ntp_servers=127.0.145.20:46749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:19:08.439286 30578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:08.439612 30578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:19:08.439771 30578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:08.447733 30578 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:19:08.447875 30578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:19:08.448004 30578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.6
I20240503 02:19:08.457126 30578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:46749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.6:39629
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.145.6
--webserver_port=46155
--tserver_master_addrs=127.0.145.62:44317
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.6
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:19:08.458423 30578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:19:08.460188 30578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:19:08.472544 30583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:08.473259 30584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:08.473870 30586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:19:08.474567 30578 server_base.cc:1034] running on GCE node
I20240503 02:19:08.475139 30578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:19:08.476002 30578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:19:08.477207 30578 hybrid_clock.cc:648] HybridClock initialized: now 1714702748477167 us; error 38 us; skew 500 ppm
I20240503 02:19:08.477522 30578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:19:08.479674 30578 webserver.cc:460] Webserver started at http://127.0.145.6:46155/ using document root <none> and password file <none>
I20240503 02:19:08.480469 30578 fs_manager.cc:359] Metadata directory not provided
I20240503 02:19:08.480592 30578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:19:08.486086 30578 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240503 02:19:08.488776 30592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:08.490046 30578 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:19:08.490193 30578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "58a2474aadf746cc9d50a38f1ee9f80c"
format_stamp: "Formatted at 2024-05-03 02:18:59 on dist-test-slave-gq9h"
I20240503 02:19:08.491138 30578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:19:08.514703 30578 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:19:08.515638 30578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:19:08.515911 30578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:19:08.516963 30578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:19:08.518532 30578 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:19:08.518625 30578 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:08.518749 30578 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:19:08.518810 30578 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:08.550060 30578 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.6:39629
I20240503 02:19:08.550097 30704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.6:39629 every 8 connection(s)
I20240503 02:19:08.551509 30578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:19:08.558466   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 30578
I20240503 02:19:08.559125   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.4:39727
--local_ip_for_outbound_sockets=127.0.145.4
--tserver_master_addrs=127.0.145.62:44317
--webserver_port=42005
--webserver_interface=127.0.145.4
--builtin_ntp_servers=127.0.145.20:46749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:19:08.564275 30705 heartbeater.cc:347] Connected to a master server at 127.0.145.62:44317
I20240503 02:19:08.564594 30705 heartbeater.cc:464] Registering TS with master...
I20240503 02:19:08.565418 30705 heartbeater.cc:511] Master 127.0.145.62:44317 requested a full tablet report, sending...
I20240503 02:19:08.567073 29460 ts_manager.cc:194] Re-registered known tserver with Master: 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6:39629)
I20240503 02:19:08.567919 29460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.6:41603
W20240503 02:19:08.904448 30709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:08.904744 30709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:19:08.904839 30709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:08.912670 30709 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:19:08.912820 30709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:19:08.912957 30709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.4
I20240503 02:19:08.922989 30709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:46749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.4:39727
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.145.4
--webserver_port=42005
--tserver_master_addrs=127.0.145.62:44317
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.4
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:19:08.924458 30709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:19:08.926223 30709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:19:08.938203 30715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:08.938783 30717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:08.939536 30714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:19:08.940222 30709 server_base.cc:1034] running on GCE node
I20240503 02:19:08.940869 30709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:19:08.941828 30709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:19:08.943015 30709 hybrid_clock.cc:648] HybridClock initialized: now 1714702748942960 us; error 46 us; skew 500 ppm
I20240503 02:19:08.943328 30709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:19:08.945490 30709 webserver.cc:460] Webserver started at http://127.0.145.4:42005/ using document root <none> and password file <none>
I20240503 02:19:08.946208 30709 fs_manager.cc:359] Metadata directory not provided
I20240503 02:19:08.946331 30709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:19:08.951483 30709 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:19:08.954038 30723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:08.955297 30709 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240503 02:19:08.955442 30709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "319ccec294484fe9ac61b2e718311b4c"
format_stamp: "Formatted at 2024-05-03 02:18:58 on dist-test-slave-gq9h"
I20240503 02:19:08.956202 30709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:19:08.990265 30709 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:19:08.991266 30709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:19:08.991518 30709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:19:08.992386 30709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:19:08.993991 30709 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:19:08.994082 30709 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:08.994204 30709 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:19:08.994258 30709 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:09.032325 30709 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.4:39727
I20240503 02:19:09.032372 30835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.4:39727 every 8 connection(s)
I20240503 02:19:09.033686 30709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:19:09.039343   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 30709
I20240503 02:19:09.039958   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.5:35793
--local_ip_for_outbound_sockets=127.0.145.5
--tserver_master_addrs=127.0.145.62:44317
--webserver_port=46513
--webserver_interface=127.0.145.5
--builtin_ntp_servers=127.0.145.20:46749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:19:09.046303 30836 heartbeater.cc:347] Connected to a master server at 127.0.145.62:44317
I20240503 02:19:09.046663 30836 heartbeater.cc:464] Registering TS with master...
I20240503 02:19:09.047688 30836 heartbeater.cc:511] Master 127.0.145.62:44317 requested a full tablet report, sending...
I20240503 02:19:09.049341 29460 ts_manager.cc:194] Re-registered known tserver with Master: 319ccec294484fe9ac61b2e718311b4c (127.0.145.4:39727)
I20240503 02:19:09.050436 29460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.4:46529
W20240503 02:19:09.375597 30839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:09.375903 30839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:19:09.376000 30839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:09.383432 30839 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:19:09.383569 30839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:19:09.383700 30839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.5
I20240503 02:19:09.392877 30839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:46749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.5:35793
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.145.5
--webserver_port=46513
--tserver_master_addrs=127.0.145.62:44317
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.5
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:19:09.394207 30839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:19:09.396078 30839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:19:09.408603 30845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:09.409173 30847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:09.409749 30844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:19:09.410370 30839 server_base.cc:1034] running on GCE node
I20240503 02:19:09.411052 30839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:19:09.412006 30839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:19:09.413210 30839 hybrid_clock.cc:648] HybridClock initialized: now 1714702749413164 us; error 35 us; skew 500 ppm
I20240503 02:19:09.413519 30839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:19:09.415812 30839 webserver.cc:460] Webserver started at http://127.0.145.5:46513/ using document root <none> and password file <none>
I20240503 02:19:09.416564 30839 fs_manager.cc:359] Metadata directory not provided
I20240503 02:19:09.416692 30839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:19:09.422237 30839 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240503 02:19:09.425115 30853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:09.426570 30839 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:19:09.426754 30839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "209bba2a25244371943e842bdf9267cd"
format_stamp: "Formatted at 2024-05-03 02:18:59 on dist-test-slave-gq9h"
I20240503 02:19:09.427532 30839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:19:09.443496 30839 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:19:09.444270 30839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:19:09.444514 30839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:19:09.445407 30839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:19:09.446888 30839 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:19:09.446960 30839 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:09.447062 30839 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:19:09.447104 30839 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:09.475658 30839 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.5:35793
I20240503 02:19:09.475693 30965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.5:35793 every 8 connection(s)
I20240503 02:19:09.477118 30839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:19:09.480993   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 30839
I20240503 02:19:09.489464 30966 heartbeater.cc:347] Connected to a master server at 127.0.145.62:44317
I20240503 02:19:09.489768 30966 heartbeater.cc:464] Registering TS with master...
I20240503 02:19:09.490552 30966 heartbeater.cc:511] Master 127.0.145.62:44317 requested a full tablet report, sending...
I20240503 02:19:09.492138 29460 ts_manager.cc:194] Re-registered known tserver with Master: 209bba2a25244371943e842bdf9267cd (127.0.145.5:35793)
I20240503 02:19:09.492920 29460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.5:48345
I20240503 02:19:09.570690 30705 heartbeater.cc:503] Master 127.0.145.62:44317 was elected leader, sending a full tablet report...
I20240503 02:19:09.987896 30771 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:09.988045 30901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:09.989531 29586 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:10.006104 29849 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:10.016492 29716 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:10.042826 30640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:10.052996 30836 heartbeater.cc:503] Master 127.0.145.62:44317 was elected leader, sending a full tablet report...
I20240503 02:19:10.200035 30771 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:10.206837 29586 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:10.209451 30901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:10.223636 30640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:10.224187 29716 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:10.256534 29849 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:10.418037 29586 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:10.423492 30901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:10.437068 30640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:10.445324 30771 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:10.448905 29716 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:10.460467 29849 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:10.495465 30966 heartbeater.cc:503] Master 127.0.145.62:44317 was elected leader, sending a full tablet report...
I20240503 02:19:10.619315 30901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:10.620101 29586 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:10.648654 30771 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:10.655565 30640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:10.656472 29849 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:10.677906 29716 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:10.754467 29738 consensus_queue.cc:237] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } }
I20240503 02:19:10.758011 29869 raft_consensus.cc:1261] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 99f937fbfb7c49f2afe278eaa73edbba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:10.758682 29606 raft_consensus.cc:1261] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Refusing update from remote peer 99f937fbfb7c49f2afe278eaa73edbba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:10.758684 31117 consensus_queue.cc:1035] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:10.759326 31116 consensus_queue.cc:1035] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:10.763526 31116 raft_consensus.cc:2932] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 209bba2a25244371943e842bdf9267cd (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } } }
I20240503 02:19:10.765338 29869 raft_consensus.cc:2932] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 209bba2a25244371943e842bdf9267cd (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } } }
W20240503 02:19:10.766104 29672 consensus_peers.cc:485] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba -> Peer 209bba2a25244371943e842bdf9267cd (127.0.145.5:35793): Couldn't send request to peer 209bba2a25244371943e842bdf9267cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e14a9b5731446669effe472c442cda1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:10.765756 29606 raft_consensus.cc:2932] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 209bba2a25244371943e842bdf9267cd (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } } }
I20240503 02:19:10.770354 29461 catalog_manager.cc:5484] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba reported cstate change: config changed from index -1 to 4, NON_VOTER 209bba2a25244371943e842bdf9267cd (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "99f937fbfb7c49f2afe278eaa73edbba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:10.783906 29606 consensus_queue.cc:237] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } }
I20240503 02:19:10.786840 29738 raft_consensus.cc:1261] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:10.787446 30520 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:10.790326 30520 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
I20240503 02:19:10.790796 29738 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
I20240503 02:19:10.798033 29457 catalog_manager.cc:5484] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba reported cstate change: config changed from index -1 to 4, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
I20240503 02:19:10.807560 29869 raft_consensus.cc:1261] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:10.809142 31140 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:10.812184 29606 consensus_queue.cc:237] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } }
I20240503 02:19:10.815558 29869 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
I20240503 02:19:10.816527 29738 raft_consensus.cc:1261] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:10.817322 29868 raft_consensus.cc:1261] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:10.817373 30520 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:19:10.817978 29541 consensus_peers.cc:485] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc -> Peer 319ccec294484fe9ac61b2e718311b4c (127.0.145.4:39727): Couldn't send request to peer 319ccec294484fe9ac61b2e718311b4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f4fdecce7bd47b9a92d66a775211292. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:10.818535 30520 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:10.822089 31134 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
I20240503 02:19:10.822665 29868 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
W20240503 02:19:10.827890 29541 consensus_peers.cc:485] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc -> Peer 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6:39629): Couldn't send request to peer 58a2474aadf746cc9d50a38f1ee9f80c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 548b1357845a405aad60446d2c5d120c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:10.828509 29457 catalog_manager.cc:5484] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 reported cstate change: config changed from index -1 to 4, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
I20240503 02:19:10.838032 29738 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
I20240503 02:19:10.844326 29738 consensus_queue.cc:237] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } }
I20240503 02:19:10.846590 29606 raft_consensus.cc:1261] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Refusing update from remote peer 99f937fbfb7c49f2afe278eaa73edbba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:10.846685 29869 raft_consensus.cc:1261] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 99f937fbfb7c49f2afe278eaa73edbba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:19:10.846740 29672 consensus_peers.cc:485] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba -> Peer 209bba2a25244371943e842bdf9267cd (127.0.145.5:35793): Couldn't send request to peer 209bba2a25244371943e842bdf9267cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91f62fd5389d4ee7ac179ed61ecd858a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:10.847172 30361 consensus_queue.cc:1035] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:10.847527 31116 consensus_queue.cc:1035] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:10.850322 31117 raft_consensus.cc:2932] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 209bba2a25244371943e842bdf9267cd (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } } }
I20240503 02:19:10.851017 29606 raft_consensus.cc:2932] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 209bba2a25244371943e842bdf9267cd (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } } }
I20240503 02:19:10.851318 29869 raft_consensus.cc:2932] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 209bba2a25244371943e842bdf9267cd (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } } }
I20240503 02:19:10.855296 29460 catalog_manager.cc:5484] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba reported cstate change: config changed from index -1 to 4, NON_VOTER 209bba2a25244371943e842bdf9267cd (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "99f937fbfb7c49f2afe278eaa73edbba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:10.868934 29606 consensus_queue.cc:237] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } }
I20240503 02:19:10.871201 29738 raft_consensus.cc:1261] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:10.871270 29869 raft_consensus.cc:1261] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:19:10.871385 29541 consensus_peers.cc:485] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc -> Peer 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6:39629): Couldn't send request to peer 58a2474aadf746cc9d50a38f1ee9f80c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b22e23ebdee48358f75d140d3681800. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:10.871750 31134 consensus_queue.cc:1035] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:10.872392 31134 consensus_queue.cc:1035] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:10.874533 31131 raft_consensus.cc:2932] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
I20240503 02:19:10.875263 29738 raft_consensus.cc:2932] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
I20240503 02:19:10.880213 29869 raft_consensus.cc:2932] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
I20240503 02:19:10.880589 29460 catalog_manager.cc:5484] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba reported cstate change: config changed from index -1 to 4, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
I20240503 02:19:10.893612 29606 consensus_queue.cc:237] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } }
I20240503 02:19:10.895974 29738 raft_consensus.cc:1261] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:19:10.896252 29541 consensus_peers.cc:485] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc -> Peer 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6:39629): Couldn't send request to peer 58a2474aadf746cc9d50a38f1ee9f80c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e13aebe3d3649dda2b2a4f157c08dbc. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:10.896229 29869 raft_consensus.cc:1261] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:10.896656 31140 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:10.896941 31131 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:10.900030 31134 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
I20240503 02:19:10.900784 29738 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
I20240503 02:19:10.900858 29869 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
I20240503 02:19:10.904620 29461 catalog_manager.cc:5484] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: config changed from index -1 to 4, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:11.215623 31227 ts_tablet_manager.cc:902] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd: Initiating tablet copy from peer 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371)
I20240503 02:19:11.217689 31227 tablet_copy_client.cc:308] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:37371
I20240503 02:19:11.225267 29758 tablet_copy_service.cc:140] P 99f937fbfb7c49f2afe278eaa73edbba: Received BeginTabletCopySession request for tablet 5e14a9b5731446669effe472c442cda1 from peer 209bba2a25244371943e842bdf9267cd ({username='slave'} at 127.0.145.5:55561)
I20240503 02:19:11.225556 29758 tablet_copy_service.cc:161] P 99f937fbfb7c49f2afe278eaa73edbba: Beginning new tablet copy session on tablet 5e14a9b5731446669effe472c442cda1 from peer 209bba2a25244371943e842bdf9267cd at {username='slave'} at 127.0.145.5:55561: session id = 209bba2a25244371943e842bdf9267cd-5e14a9b5731446669effe472c442cda1
I20240503 02:19:11.229493 29758 tablet_copy_source_session.cc:205] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:11.235297 31227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e14a9b5731446669effe472c442cda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:11.243167 31235 ts_tablet_manager.cc:902] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd: Initiating tablet copy from peer 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371)
I20240503 02:19:11.247213 31227 tablet_copy_client.cc:783] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:11.247604 31227 tablet_copy_client.cc:647] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:11.247988 31235 tablet_copy_client.cc:308] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:37371
I20240503 02:19:11.248903 29758 tablet_copy_service.cc:140] P 99f937fbfb7c49f2afe278eaa73edbba: Received BeginTabletCopySession request for tablet 91f62fd5389d4ee7ac179ed61ecd858a from peer 209bba2a25244371943e842bdf9267cd ({username='slave'} at 127.0.145.5:55561)
I20240503 02:19:11.249109 29758 tablet_copy_service.cc:161] P 99f937fbfb7c49f2afe278eaa73edbba: Beginning new tablet copy session on tablet 91f62fd5389d4ee7ac179ed61ecd858a from peer 209bba2a25244371943e842bdf9267cd at {username='slave'} at 127.0.145.5:55561: session id = 209bba2a25244371943e842bdf9267cd-91f62fd5389d4ee7ac179ed61ecd858a
I20240503 02:19:11.250670 31227 tablet_copy_client.cc:523] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:11.251399 29758 tablet_copy_source_session.cc:205] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:11.252514 31235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f62fd5389d4ee7ac179ed61ecd858a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:11.255637 31227 tablet_bootstrap.cc:492] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd: Bootstrap starting.
I20240503 02:19:11.257278 31235 tablet_copy_client.cc:783] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:11.257513 31235 tablet_copy_client.cc:647] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:11.259433 31235 tablet_copy_client.cc:523] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:11.261996 31235 tablet_bootstrap.cc:492] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd: Bootstrap starting.
I20240503 02:19:11.266427 31227 log.cc:824] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd: Log is configured to *not* fsync() on all Append() calls
I20240503 02:19:11.298494 31235 tablet_bootstrap.cc:492] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:11.299060 31235 tablet_bootstrap.cc:492] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd: Bootstrap complete.
I20240503 02:19:11.299520 31235 ts_tablet_manager.cc:1367] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd: Time spent bootstrapping tablet: real 0.038s	user 0.034s	sys 0.004s
I20240503 02:19:11.299656 31227 tablet_bootstrap.cc:492] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:11.300110 31227 tablet_bootstrap.cc:492] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd: Bootstrap complete.
I20240503 02:19:11.300349 31227 ts_tablet_manager.cc:1367] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd: Time spent bootstrapping tablet: real 0.045s	user 0.035s	sys 0.008s
I20240503 02:19:11.304896 31235 raft_consensus.cc:348] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } }
I20240503 02:19:11.304895 31227 raft_consensus.cc:348] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } }
I20240503 02:19:11.305425 31227 raft_consensus.cc:729] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 209bba2a25244371943e842bdf9267cd, State: Initialized, Role: LEARNER
I20240503 02:19:11.305433 31235 raft_consensus.cc:729] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 209bba2a25244371943e842bdf9267cd, State: Initialized, Role: LEARNER
I20240503 02:19:11.306092 31235 consensus_queue.cc:260] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } }
I20240503 02:19:11.306099 31227 consensus_queue.cc:260] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } }
I20240503 02:19:11.307890 31235 ts_tablet_manager.cc:1398] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20240503 02:19:11.307889 31227 ts_tablet_manager.cc:1398] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20240503 02:19:11.308919 29757 tablet_copy_service.cc:342] P 99f937fbfb7c49f2afe278eaa73edbba: Request end of tablet copy session 209bba2a25244371943e842bdf9267cd-91f62fd5389d4ee7ac179ed61ecd858a received from {username='slave'} at 127.0.145.5:55561
I20240503 02:19:11.309074 29757 tablet_copy_service.cc:433] P 99f937fbfb7c49f2afe278eaa73edbba: ending tablet copy session 209bba2a25244371943e842bdf9267cd-91f62fd5389d4ee7ac179ed61ecd858a on tablet 91f62fd5389d4ee7ac179ed61ecd858a with peer 209bba2a25244371943e842bdf9267cd
I20240503 02:19:11.309531 29758 tablet_copy_service.cc:342] P 99f937fbfb7c49f2afe278eaa73edbba: Request end of tablet copy session 209bba2a25244371943e842bdf9267cd-5e14a9b5731446669effe472c442cda1 received from {username='slave'} at 127.0.145.5:55561
I20240503 02:19:11.309650 29758 tablet_copy_service.cc:433] P 99f937fbfb7c49f2afe278eaa73edbba: ending tablet copy session 209bba2a25244371943e842bdf9267cd-5e14a9b5731446669effe472c442cda1 on tablet 5e14a9b5731446669effe472c442cda1 with peer 209bba2a25244371943e842bdf9267cd
I20240503 02:19:11.377959 31244 ts_tablet_manager.cc:902] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c: Initiating tablet copy from peer 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557)
I20240503 02:19:11.379729 31244 tablet_copy_client.cc:308] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:46557
I20240503 02:19:11.386687 29626 tablet_copy_service.cc:140] P 07bd09ad30cc4932bfff380344c8e9fc: Received BeginTabletCopySession request for tablet 1b22e23ebdee48358f75d140d3681800 from peer 58a2474aadf746cc9d50a38f1ee9f80c ({username='slave'} at 127.0.145.6:45531)
I20240503 02:19:11.386935 29626 tablet_copy_service.cc:161] P 07bd09ad30cc4932bfff380344c8e9fc: Beginning new tablet copy session on tablet 1b22e23ebdee48358f75d140d3681800 from peer 58a2474aadf746cc9d50a38f1ee9f80c at {username='slave'} at 127.0.145.6:45531: session id = 58a2474aadf746cc9d50a38f1ee9f80c-1b22e23ebdee48358f75d140d3681800
I20240503 02:19:11.389075 31250 ts_tablet_manager.cc:902] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: Initiating tablet copy from peer 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557)
I20240503 02:19:11.390372 29626 tablet_copy_source_session.cc:205] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:11.392568 31250 tablet_copy_client.cc:308] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:46557
I20240503 02:19:11.393329 29626 tablet_copy_service.cc:140] P 07bd09ad30cc4932bfff380344c8e9fc: Received BeginTabletCopySession request for tablet 548b1357845a405aad60446d2c5d120c from peer 58a2474aadf746cc9d50a38f1ee9f80c ({username='slave'} at 127.0.145.6:45531)
I20240503 02:19:11.393507 29626 tablet_copy_service.cc:161] P 07bd09ad30cc4932bfff380344c8e9fc: Beginning new tablet copy session on tablet 548b1357845a405aad60446d2c5d120c from peer 58a2474aadf746cc9d50a38f1ee9f80c at {username='slave'} at 127.0.145.6:45531: session id = 58a2474aadf746cc9d50a38f1ee9f80c-548b1357845a405aad60446d2c5d120c
I20240503 02:19:11.396301 29626 tablet_copy_source_session.cc:205] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:11.396370 31244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b22e23ebdee48358f75d140d3681800. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:11.397629 31250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548b1357845a405aad60446d2c5d120c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:11.403560 31255 ts_tablet_manager.cc:902] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c: Initiating tablet copy from peer 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557)
I20240503 02:19:11.405344 31255 tablet_copy_client.cc:308] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:46557
I20240503 02:19:11.408959 31250 tablet_copy_client.cc:783] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:11.409215 31244 tablet_copy_client.cc:783] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:11.409405 31250 tablet_copy_client.cc:647] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:11.409463 31244 tablet_copy_client.cc:647] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:11.412833 29625 tablet_copy_service.cc:140] P 07bd09ad30cc4932bfff380344c8e9fc: Received BeginTabletCopySession request for tablet 6f4fdecce7bd47b9a92d66a775211292 from peer 319ccec294484fe9ac61b2e718311b4c ({username='slave'} at 127.0.145.4:48669)
I20240503 02:19:11.413022 29625 tablet_copy_service.cc:161] P 07bd09ad30cc4932bfff380344c8e9fc: Beginning new tablet copy session on tablet 6f4fdecce7bd47b9a92d66a775211292 from peer 319ccec294484fe9ac61b2e718311b4c at {username='slave'} at 127.0.145.4:48669: session id = 319ccec294484fe9ac61b2e718311b4c-6f4fdecce7bd47b9a92d66a775211292
I20240503 02:19:11.413159 31244 tablet_copy_client.cc:523] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:11.413162 31250 tablet_copy_client.cc:523] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:11.416415 29625 tablet_copy_source_session.cc:205] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:11.416911 31244 tablet_bootstrap.cc:492] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c: Bootstrap starting.
I20240503 02:19:11.416932 31250 tablet_bootstrap.cc:492] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: Bootstrap starting.
I20240503 02:19:11.421923 31255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4fdecce7bd47b9a92d66a775211292. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:11.429841 31244 log.cc:824] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c: Log is configured to *not* fsync() on all Append() calls
I20240503 02:19:11.439728 31255 tablet_copy_client.cc:783] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:11.440304 31255 tablet_copy_client.cc:647] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:11.443487 31255 tablet_copy_client.cc:523] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:11.448012 31255 tablet_bootstrap.cc:492] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c: Bootstrap starting.
I20240503 02:19:11.461766 31255 log.cc:824] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c: Log is configured to *not* fsync() on all Append() calls
I20240503 02:19:11.489866 31244 tablet_bootstrap.cc:492] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:11.490655 31244 tablet_bootstrap.cc:492] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c: Bootstrap complete.
I20240503 02:19:11.491258 31244 ts_tablet_manager.cc:1367] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c: Time spent bootstrapping tablet: real 0.075s	user 0.063s	sys 0.011s
I20240503 02:19:11.492045 31250 tablet_bootstrap.cc:492] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:11.492627 31250 tablet_bootstrap.cc:492] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: Bootstrap complete.
I20240503 02:19:11.492980 31250 ts_tablet_manager.cc:1367] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: Time spent bootstrapping tablet: real 0.076s	user 0.072s	sys 0.000s
I20240503 02:19:11.498348 31250 raft_consensus.cc:348] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } }
I20240503 02:19:11.498351 31244 raft_consensus.cc:348] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } }
I20240503 02:19:11.499109 31250 raft_consensus.cc:729] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58a2474aadf746cc9d50a38f1ee9f80c, State: Initialized, Role: LEARNER
I20240503 02:19:11.499167 31244 raft_consensus.cc:729] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58a2474aadf746cc9d50a38f1ee9f80c, State: Initialized, Role: LEARNER
I20240503 02:19:11.499855 31244 consensus_queue.cc:260] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } }
I20240503 02:19:11.499851 31250 consensus_queue.cc:260] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } }
I20240503 02:19:11.505712 31250 ts_tablet_manager.cc:1398] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20240503 02:19:11.513473 29625 tablet_copy_service.cc:342] P 07bd09ad30cc4932bfff380344c8e9fc: Request end of tablet copy session 58a2474aadf746cc9d50a38f1ee9f80c-548b1357845a405aad60446d2c5d120c received from {username='slave'} at 127.0.145.6:45531
I20240503 02:19:11.513680 29625 tablet_copy_service.cc:433] P 07bd09ad30cc4932bfff380344c8e9fc: ending tablet copy session 58a2474aadf746cc9d50a38f1ee9f80c-548b1357845a405aad60446d2c5d120c on tablet 548b1357845a405aad60446d2c5d120c with peer 58a2474aadf746cc9d50a38f1ee9f80c
I20240503 02:19:11.517143 31250 ts_tablet_manager.cc:902] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c: Initiating tablet copy from peer 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557)
I20240503 02:19:11.518146 31244 ts_tablet_manager.cc:1398] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c: Time spent starting tablet: real 0.027s	user 0.019s	sys 0.005s
I20240503 02:19:11.519464 29625 tablet_copy_service.cc:342] P 07bd09ad30cc4932bfff380344c8e9fc: Request end of tablet copy session 58a2474aadf746cc9d50a38f1ee9f80c-1b22e23ebdee48358f75d140d3681800 received from {username='slave'} at 127.0.145.6:45531
I20240503 02:19:11.519606 29625 tablet_copy_service.cc:433] P 07bd09ad30cc4932bfff380344c8e9fc: ending tablet copy session 58a2474aadf746cc9d50a38f1ee9f80c-1b22e23ebdee48358f75d140d3681800 on tablet 1b22e23ebdee48358f75d140d3681800 with peer 58a2474aadf746cc9d50a38f1ee9f80c
I20240503 02:19:11.533195 31255 tablet_bootstrap.cc:492] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:11.534328 31255 tablet_bootstrap.cc:492] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c: Bootstrap complete.
I20240503 02:19:11.535185 31255 ts_tablet_manager.cc:1367] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c: Time spent bootstrapping tablet: real 0.087s	user 0.057s	sys 0.020s
I20240503 02:19:11.537099 31250 tablet_copy_client.cc:308] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:46557
I20240503 02:19:11.537945 29625 tablet_copy_service.cc:140] P 07bd09ad30cc4932bfff380344c8e9fc: Received BeginTabletCopySession request for tablet 9e13aebe3d3649dda2b2a4f157c08dbc from peer 58a2474aadf746cc9d50a38f1ee9f80c ({username='slave'} at 127.0.145.6:45531)
I20240503 02:19:11.538132 29625 tablet_copy_service.cc:161] P 07bd09ad30cc4932bfff380344c8e9fc: Beginning new tablet copy session on tablet 9e13aebe3d3649dda2b2a4f157c08dbc from peer 58a2474aadf746cc9d50a38f1ee9f80c at {username='slave'} at 127.0.145.6:45531: session id = 58a2474aadf746cc9d50a38f1ee9f80c-9e13aebe3d3649dda2b2a4f157c08dbc
I20240503 02:19:11.541690 29625 tablet_copy_source_session.cc:205] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:11.541954 31255 raft_consensus.cc:348] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } }
I20240503 02:19:11.542510 31255 raft_consensus.cc:729] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 319ccec294484fe9ac61b2e718311b4c, State: Initialized, Role: LEARNER
I20240503 02:19:11.543105 31255 consensus_queue.cc:260] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } }
I20240503 02:19:11.543390 31250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e13aebe3d3649dda2b2a4f157c08dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:11.549765 31250 tablet_copy_client.cc:783] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:11.550011 31250 tablet_copy_client.cc:647] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:11.552009 31250 tablet_copy_client.cc:523] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:11.555318 31250 tablet_bootstrap.cc:492] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c: Bootstrap starting.
I20240503 02:19:11.565193 31255 ts_tablet_manager.cc:1398] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.009s
I20240503 02:19:11.567209 29625 tablet_copy_service.cc:342] P 07bd09ad30cc4932bfff380344c8e9fc: Request end of tablet copy session 319ccec294484fe9ac61b2e718311b4c-6f4fdecce7bd47b9a92d66a775211292 received from {username='slave'} at 127.0.145.4:48669
I20240503 02:19:11.567338 29625 tablet_copy_service.cc:433] P 07bd09ad30cc4932bfff380344c8e9fc: ending tablet copy session 319ccec294484fe9ac61b2e718311b4c-6f4fdecce7bd47b9a92d66a775211292 on tablet 6f4fdecce7bd47b9a92d66a775211292 with peer 319ccec294484fe9ac61b2e718311b4c
I20240503 02:19:11.607723 31250 tablet_bootstrap.cc:492] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:11.608134 31250 tablet_bootstrap.cc:492] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c: Bootstrap complete.
I20240503 02:19:11.608376 31250 ts_tablet_manager.cc:1367] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c: Time spent bootstrapping tablet: real 0.053s	user 0.041s	sys 0.009s
I20240503 02:19:11.609061 31250 raft_consensus.cc:348] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } }
I20240503 02:19:11.609258 31250 raft_consensus.cc:729] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58a2474aadf746cc9d50a38f1ee9f80c, State: Initialized, Role: LEARNER
I20240503 02:19:11.609472 31250 consensus_queue.cc:260] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } }
I20240503 02:19:11.610145 31250 ts_tablet_manager.cc:1398] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:19:11.611105 29625 tablet_copy_service.cc:342] P 07bd09ad30cc4932bfff380344c8e9fc: Request end of tablet copy session 58a2474aadf746cc9d50a38f1ee9f80c-9e13aebe3d3649dda2b2a4f157c08dbc received from {username='slave'} at 127.0.145.6:45531
I20240503 02:19:11.611214 29625 tablet_copy_service.cc:433] P 07bd09ad30cc4932bfff380344c8e9fc: ending tablet copy session 58a2474aadf746cc9d50a38f1ee9f80c-9e13aebe3d3649dda2b2a4f157c08dbc on tablet 9e13aebe3d3649dda2b2a4f157c08dbc with peer 58a2474aadf746cc9d50a38f1ee9f80c
I20240503 02:19:11.671510 30921 raft_consensus.cc:1203] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:11.726959 30921 raft_consensus.cc:1203] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:11.776355 30660 raft_consensus.cc:1203] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:11.874317 30791 raft_consensus.cc:1203] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:11.932564 30660 raft_consensus.cc:1203] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:12.000249 30660 raft_consensus.cc:1203] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:12.064283 31304 raft_consensus.cc:1050] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba: attempting to promote NON_VOTER 209bba2a25244371943e842bdf9267cd to VOTER
I20240503 02:19:12.065361 31304 consensus_queue.cc:237] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } }
I20240503 02:19:12.067762 29869 raft_consensus.cc:1261] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 99f937fbfb7c49f2afe278eaa73edbba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.067965 30921 raft_consensus.cc:1261] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd [term 1 LEARNER]: Refusing update from remote peer 99f937fbfb7c49f2afe278eaa73edbba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.068406 29606 raft_consensus.cc:1261] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Refusing update from remote peer 99f937fbfb7c49f2afe278eaa73edbba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.068472 30361 consensus_queue.cc:1035] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.068781 31304 consensus_queue.cc:1035] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.069761 31303 consensus_queue.cc:1035] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.072719 30361 raft_consensus.cc:2932] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 209bba2a25244371943e842bdf9267cd (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.075364 30921 raft_consensus.cc:2932] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 209bba2a25244371943e842bdf9267cd (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.076752 29605 raft_consensus.cc:2932] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 209bba2a25244371943e842bdf9267cd (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.079885 29869 raft_consensus.cc:2932] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 209bba2a25244371943e842bdf9267cd (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.081102 29460 catalog_manager.cc:5484] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba reported cstate change: config changed from index 4 to 5, 209bba2a25244371943e842bdf9267cd (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99f937fbfb7c49f2afe278eaa73edbba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:12.091239 29738 consensus_queue.cc:237] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } }
I20240503 02:19:12.093210 30921 raft_consensus.cc:1261] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd [term 1 FOLLOWER]: Refusing update from remote peer 99f937fbfb7c49f2afe278eaa73edbba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:12.093418 29869 raft_consensus.cc:1261] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 99f937fbfb7c49f2afe278eaa73edbba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:12.093719 31303 consensus_queue.cc:1035] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:12.094409 31303 consensus_queue.cc:1035] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:12.096866 31304 raft_consensus.cc:2932] T 5e14a9b5731446669effe472c442cda1 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.097478 29869 raft_consensus.cc:2932] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.100067 29449 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5e14a9b5731446669effe472c442cda1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:12.100541 30921 raft_consensus.cc:2932] T 5e14a9b5731446669effe472c442cda1 P 209bba2a25244371943e842bdf9267cd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.103902 29464 catalog_manager.cc:5484] T 5e14a9b5731446669effe472c442cda1 P 6c36b0a7f8a3485f9a9d62e3e7793f25 reported cstate change: config changed from index 5 to 6, VOTER 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "99f937fbfb7c49f2afe278eaa73edbba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.108853 29586 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e14a9b5731446669effe472c442cda1 with delete_type TABLET_DATA_TOMBSTONED (TS 07bd09ad30cc4932bfff380344c8e9fc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48014
I20240503 02:19:12.111052 31355 tablet_replica.cc:330] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc: stopping tablet replica
I20240503 02:19:12.111493 31355 raft_consensus.cc:2229] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:12.111850 31355 raft_consensus.cc:2258] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:12.113864 31355 ts_tablet_manager.cc:1875] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:12.120476 31355 ts_tablet_manager.cc:1888] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:19:12.120705 31355 log.cc:1196] T 5e14a9b5731446669effe472c442cda1 P 07bd09ad30cc4932bfff380344c8e9fc: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/5e14a9b5731446669effe472c442cda1
I20240503 02:19:12.121727 29450 catalog_manager.cc:4830] TS 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557): tablet 5e14a9b5731446669effe472c442cda1 (table rebalance_test_table_1 [id=11a9093ed71046c1b45b6520ebcc4ed0]) successfully deleted
I20240503 02:19:12.197363 30361 raft_consensus.cc:1050] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba: attempting to promote NON_VOTER 209bba2a25244371943e842bdf9267cd to VOTER
I20240503 02:19:12.197963 30361 consensus_queue.cc:237] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } }
I20240503 02:19:12.200100 29869 raft_consensus.cc:1261] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 99f937fbfb7c49f2afe278eaa73edbba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.200116 30921 raft_consensus.cc:1261] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd [term 1 LEARNER]: Refusing update from remote peer 99f937fbfb7c49f2afe278eaa73edbba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.200276 29605 raft_consensus.cc:1261] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Refusing update from remote peer 99f937fbfb7c49f2afe278eaa73edbba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.200863 30361 consensus_queue.cc:1035] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.201160 31353 consensus_queue.cc:1035] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.201431 31354 consensus_queue.cc:1035] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.205716 31353 raft_consensus.cc:2932] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 209bba2a25244371943e842bdf9267cd (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.206554 29869 raft_consensus.cc:2932] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 209bba2a25244371943e842bdf9267cd (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.209290 30921 raft_consensus.cc:2932] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 209bba2a25244371943e842bdf9267cd (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.213591 29461 catalog_manager.cc:5484] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25 reported cstate change: config changed from index 4 to 5, 209bba2a25244371943e842bdf9267cd (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99f937fbfb7c49f2afe278eaa73edbba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.218822 29738 consensus_queue.cc:237] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } }
I20240503 02:19:12.220783 30921 raft_consensus.cc:1261] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd [term 1 FOLLOWER]: Refusing update from remote peer 99f937fbfb7c49f2afe278eaa73edbba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:12.220927 29869 raft_consensus.cc:1261] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 99f937fbfb7c49f2afe278eaa73edbba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:12.221369 31353 consensus_queue.cc:1035] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:12.222151 31353 consensus_queue.cc:1035] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:12.223593 31354 raft_consensus.cc:2932] T 91f62fd5389d4ee7ac179ed61ecd858a P 99f937fbfb7c49f2afe278eaa73edbba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.224162 30921 raft_consensus.cc:2932] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.228065 29449 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 91f62fd5389d4ee7ac179ed61ecd858a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:12.228183 29460 catalog_manager.cc:5484] T 91f62fd5389d4ee7ac179ed61ecd858a P 209bba2a25244371943e842bdf9267cd reported cstate change: config changed from index 5 to 6, VOTER 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "99f937fbfb7c49f2afe278eaa73edbba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.230510 29869 raft_consensus.cc:2932] T 91f62fd5389d4ee7ac179ed61ecd858a P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:12.232000 29586 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f62fd5389d4ee7ac179ed61ecd858a with delete_type TABLET_DATA_TOMBSTONED (TS 07bd09ad30cc4932bfff380344c8e9fc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48014
I20240503 02:19:12.232439 31355 tablet_replica.cc:330] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc: stopping tablet replica
I20240503 02:19:12.232645 31355 raft_consensus.cc:2229] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:12.232846 31355 pending_rounds.cc:62] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc: Trying to abort 1 pending ops.
I20240503 02:19:12.232932 31355 pending_rounds.cc:68] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023422473001398272 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "91f62fd5389d4ee7ac179ed61ecd858a" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } } }
I20240503 02:19:12.233264 31355 raft_consensus.cc:2894] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 02:19:12.233357 31355 raft_consensus.cc:2866] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:19:12.233436 31355 raft_consensus.cc:2258] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:12.234504 31355 ts_tablet_manager.cc:1875] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:12.239306 31355 ts_tablet_manager.cc:1888] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:19:12.239475 31355 log.cc:1196] T 91f62fd5389d4ee7ac179ed61ecd858a P 07bd09ad30cc4932bfff380344c8e9fc: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/91f62fd5389d4ee7ac179ed61ecd858a
I20240503 02:19:12.240968 29450 catalog_manager.cc:4830] TS 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557): tablet 91f62fd5389d4ee7ac179ed61ecd858a (table rebalance_test_table_0 [id=dab45517a1ea4baa80835e9059e98803]) successfully deleted
I20240503 02:19:12.270164 31134 raft_consensus.cc:1050] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc: attempting to promote NON_VOTER 319ccec294484fe9ac61b2e718311b4c to VOTER
I20240503 02:19:12.270933 31134 consensus_queue.cc:237] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } }
I20240503 02:19:12.273151 29738 raft_consensus.cc:1261] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.273177 29869 raft_consensus.cc:1261] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.273303 30791 raft_consensus.cc:1261] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.273705 31134 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.273996 31406 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.274295 31140 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.278091 31140 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:12.279065 30791 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:12.281040 29869 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:12.282274 29738 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:12.282992 29460 catalog_manager.cc:5484] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:12.288561 29605 consensus_queue.cc:237] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } }
I20240503 02:19:12.290033 29738 raft_consensus.cc:1261] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:12.290439 30791 raft_consensus.cc:1261] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:12.290509 31140 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:12.290906 31134 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:12.292698 31140 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:12.293471 30791 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:12.293548 29738 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:12.296339 29450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6f4fdecce7bd47b9a92d66a775211292 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:12.297500 29461 catalog_manager.cc:5484] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c reported cstate change: config changed from index 5 to 6, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:12.299578 31406 raft_consensus.cc:1050] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc: attempting to promote NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c to VOTER
I20240503 02:19:12.300299 31406 consensus_queue.cc:237] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } }
I20240503 02:19:12.301820 29849 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f4fdecce7bd47b9a92d66a775211292 with delete_type TABLET_DATA_TOMBSTONED (TS 6c36b0a7f8a3485f9a9d62e3e7793f25 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38302
I20240503 02:19:12.302837 31411 tablet_replica.cc:330] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: stopping tablet replica
I20240503 02:19:12.303349 31411 raft_consensus.cc:2229] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:12.303349 29738 raft_consensus.cc:1261] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.303452 29869 raft_consensus.cc:1261] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.303691 31411 raft_consensus.cc:2258] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:12.303718 30660 raft_consensus.cc:1261] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 LEARNER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.303997 31140 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.304322 31406 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.304595 31134 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.305544 31411 ts_tablet_manager.cc:1875] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:12.314893 31411 ts_tablet_manager.cc:1888] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:19:12.315227 31411 log.cc:1196] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/6f4fdecce7bd47b9a92d66a775211292
I20240503 02:19:12.317009 29451 catalog_manager.cc:4830] TS 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133): tablet 6f4fdecce7bd47b9a92d66a775211292 (table rebalance_test_table_2 [id=a25280dd90f64d31a487b2c771383e8a]) successfully deleted
I20240503 02:19:12.316987 31131 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.317734 29868 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.320979 29738 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.322978 29461 catalog_manager.cc:5484] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:12.323439 30660 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.328030 29605 consensus_queue.cc:237] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } }
I20240503 02:19:12.329535 30660 raft_consensus.cc:1261] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:12.329703 29868 raft_consensus.cc:1261] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:12.330054 31131 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:12.330348 31134 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:12.332536 31134 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.333181 30660 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.333274 29868 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.337117 29450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 548b1357845a405aad60446d2c5d120c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:12.338106 29457 catalog_manager.cc:5484] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c reported cstate change: config changed from index 5 to 6, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.341455 29716 tablet_service.cc:1514] Processing DeleteTablet for tablet 548b1357845a405aad60446d2c5d120c with delete_type TABLET_DATA_TOMBSTONED (TS 99f937fbfb7c49f2afe278eaa73edbba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48100
I20240503 02:19:12.342389 31415 tablet_replica.cc:330] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba: stopping tablet replica
I20240503 02:19:12.342865 31415 raft_consensus.cc:2229] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:12.343129 31415 raft_consensus.cc:2258] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:12.344513 31415 ts_tablet_manager.cc:1875] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:12.349278 31415 ts_tablet_manager.cc:1888] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:19:12.349432 31415 log.cc:1196] T 548b1357845a405aad60446d2c5d120c P 99f937fbfb7c49f2afe278eaa73edbba: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/548b1357845a405aad60446d2c5d120c
I20240503 02:19:12.350334 29449 catalog_manager.cc:4830] TS 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371): tablet 548b1357845a405aad60446d2c5d120c (table rebalance_test_table_0 [id=dab45517a1ea4baa80835e9059e98803]) successfully deleted
I20240503 02:19:12.359908 31134 raft_consensus.cc:1050] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc: attempting to promote NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c to VOTER
I20240503 02:19:12.360648 31134 consensus_queue.cc:237] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } }
I20240503 02:19:12.362891 29738 raft_consensus.cc:1261] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.362928 30660 raft_consensus.cc:1261] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 LEARNER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.363016 29868 raft_consensus.cc:1261] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.363452 31134 consensus_queue.cc:1035] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.363734 31140 consensus_queue.cc:1035] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.364113 31131 consensus_queue.cc:1035] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.367856 31140 raft_consensus.cc:2932] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.371248 29738 raft_consensus.cc:2932] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.372062 30660 raft_consensus.cc:2932] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.372277 29461 catalog_manager.cc:5484] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:12.372283 29868 raft_consensus.cc:2932] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.381234 29606 consensus_queue.cc:237] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } }
I20240503 02:19:12.382884 30660 raft_consensus.cc:1261] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:12.383219 29868 raft_consensus.cc:1261] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:12.383436 31406 consensus_queue.cc:1035] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:12.383761 31134 consensus_queue.cc:1035] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:12.385813 31140 raft_consensus.cc:2932] T 1b22e23ebdee48358f75d140d3681800 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.386513 30660 raft_consensus.cc:2932] T 1b22e23ebdee48358f75d140d3681800 P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.386878 29868 raft_consensus.cc:2932] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.391106 29450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b22e23ebdee48358f75d140d3681800 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:12.391227 29460 catalog_manager.cc:5484] T 1b22e23ebdee48358f75d140d3681800 P 6c36b0a7f8a3485f9a9d62e3e7793f25 reported cstate change: config changed from index 5 to 6, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.395097 29716 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b22e23ebdee48358f75d140d3681800 with delete_type TABLET_DATA_TOMBSTONED (TS 99f937fbfb7c49f2afe278eaa73edbba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48100
I20240503 02:19:12.395500 31415 tablet_replica.cc:330] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba: stopping tablet replica
I20240503 02:19:12.395735 31415 raft_consensus.cc:2229] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:12.396093 31415 raft_consensus.cc:2258] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:12.397158 31415 ts_tablet_manager.cc:1875] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:12.403367 31415 ts_tablet_manager.cc:1888] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:19:12.403527 31415 log.cc:1196] T 1b22e23ebdee48358f75d140d3681800 P 99f937fbfb7c49f2afe278eaa73edbba: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/1b22e23ebdee48358f75d140d3681800
I20240503 02:19:12.404269 29449 catalog_manager.cc:4830] TS 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371): tablet 1b22e23ebdee48358f75d140d3681800 (table rebalance_test_table_1 [id=11a9093ed71046c1b45b6520ebcc4ed0]) successfully deleted
I20240503 02:19:12.498634 31134 raft_consensus.cc:1050] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc: attempting to promote NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c to VOTER
I20240503 02:19:12.499447 31134 consensus_queue.cc:237] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } }
I20240503 02:19:12.501629 30660 raft_consensus.cc:1261] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 LEARNER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.501681 29738 raft_consensus.cc:1261] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.501725 29868 raft_consensus.cc:1261] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:12.502187 31406 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.502491 31131 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.502784 31134 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:12.511978 31406 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.513216 29738 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.514264 30660 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.516335 29868 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.518554 29457 catalog_manager.cc:5484] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba reported cstate change: config changed from index 4 to 5, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.524468 29606 consensus_queue.cc:237] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } }
I20240503 02:19:12.526160 29868 raft_consensus.cc:1261] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:12.526186 30660 raft_consensus.cc:1261] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:12.526686 31406 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:12.527033 31140 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:12.529086 31140 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.529842 30660 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.529953 29868 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:12.533619 29450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e13aebe3d3649dda2b2a4f157c08dbc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:12.534370 29461 catalog_manager.cc:5484] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: config changed from index 5 to 6, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:12.538049 29716 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e13aebe3d3649dda2b2a4f157c08dbc with delete_type TABLET_DATA_TOMBSTONED (TS 99f937fbfb7c49f2afe278eaa73edbba not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48100
I20240503 02:19:12.538381 31415 tablet_replica.cc:330] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: stopping tablet replica
I20240503 02:19:12.538586 31415 raft_consensus.cc:2229] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:12.538908 31415 raft_consensus.cc:2258] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:12.539748 31415 ts_tablet_manager.cc:1875] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:12.543880 31415 ts_tablet_manager.cc:1888] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:19:12.543972 31415 log.cc:1196] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/9e13aebe3d3649dda2b2a4f157c08dbc
I20240503 02:19:12.544543 29449 catalog_manager.cc:4830] TS 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371): tablet 9e13aebe3d3649dda2b2a4f157c08dbc (table rebalance_test_table_0 [id=dab45517a1ea4baa80835e9059e98803]) successfully deleted
I20240503 02:19:12.756961 29586 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:12.777475 30771 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:12.778194 30901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:12.784695 30640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:12.798687 29716 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:12.810825 29849 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:12.873143 29868 consensus_queue.cc:237] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } }
I20240503 02:19:12.875761 29606 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Refusing update from remote peer 6c36b0a7f8a3485f9a9d62e3e7793f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:12.875931 29738 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 6c36b0a7f8a3485f9a9d62e3e7793f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:12.876359 31497 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:12.876669 30549 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:12.879782 30549 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
I20240503 02:19:12.880332 29738 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
I20240503 02:19:12.887421 29460 catalog_manager.cc:5484] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba reported cstate change: config changed from index -1 to 4, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New cstate: current_term: 1 leader_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
W20240503 02:19:12.888332 29804 consensus_peers.cc:485] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 -> Peer 319ccec294484fe9ac61b2e718311b4c (127.0.145.4:39727): Couldn't send request to peer 319ccec294484fe9ac61b2e718311b4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f83faf506c3e49d4ade5f48881d9d1a7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:12.894327 29606 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
I20240503 02:19:12.900075 29868 consensus_queue.cc:237] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } }
I20240503 02:19:12.902635 29738 raft_consensus.cc:1261] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 6c36b0a7f8a3485f9a9d62e3e7793f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:12.902954 29606 raft_consensus.cc:1261] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Refusing update from remote peer 6c36b0a7f8a3485f9a9d62e3e7793f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:12.903483 31506 consensus_queue.cc:1035] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:12.903788 30549 consensus_queue.cc:1035] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:12.907138 31497 raft_consensus.cc:2932] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 209bba2a25244371943e842bdf9267cd (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } } }
I20240503 02:19:12.907825 29736 raft_consensus.cc:2932] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 209bba2a25244371943e842bdf9267cd (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } } }
W20240503 02:19:12.911772 29803 consensus_peers.cc:485] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 -> Peer 209bba2a25244371943e842bdf9267cd (127.0.145.5:35793): Couldn't send request to peer 209bba2a25244371943e842bdf9267cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 899b3fc0c13540e485d0b76417c205e4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:12.912900 29461 catalog_manager.cc:5484] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 reported cstate change: config changed from index -1 to 4, NON_VOTER 209bba2a25244371943e842bdf9267cd (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:12.914557 29606 raft_consensus.cc:2932] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 209bba2a25244371943e842bdf9267cd (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } } }
I20240503 02:19:12.927336 29606 consensus_queue.cc:237] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } }
I20240503 02:19:12.929567 29736 raft_consensus.cc:1261] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:12.929597 29868 raft_consensus.cc:1261] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:19:12.929899 29541 consensus_peers.cc:485] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc -> Peer 319ccec294484fe9ac61b2e718311b4c (127.0.145.4:39727): Couldn't send request to peer 319ccec294484fe9ac61b2e718311b4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2451cf2c781a44e0882f111cb29fa40f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:12.930142 31140 consensus_queue.cc:1035] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:12.930404 31134 consensus_queue.cc:1035] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:12.933238 31131 raft_consensus.cc:2932] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
I20240503 02:19:12.933840 29736 raft_consensus.cc:2932] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
I20240503 02:19:12.937795 29869 raft_consensus.cc:2932] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
I20240503 02:19:12.940027 29461 catalog_manager.cc:5484] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba reported cstate change: config changed from index -1 to 4, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
I20240503 02:19:13.257545 31548 ts_tablet_manager.cc:902] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c: Initiating tablet copy from peer 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:13.258809 31548 tablet_copy_client.cc:308] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:41133
I20240503 02:19:13.264861 29889 tablet_copy_service.cc:140] P 6c36b0a7f8a3485f9a9d62e3e7793f25: Received BeginTabletCopySession request for tablet f83faf506c3e49d4ade5f48881d9d1a7 from peer 319ccec294484fe9ac61b2e718311b4c ({username='slave'} at 127.0.145.4:52013)
I20240503 02:19:13.265103 29889 tablet_copy_service.cc:161] P 6c36b0a7f8a3485f9a9d62e3e7793f25: Beginning new tablet copy session on tablet f83faf506c3e49d4ade5f48881d9d1a7 from peer 319ccec294484fe9ac61b2e718311b4c at {username='slave'} at 127.0.145.4:52013: session id = 319ccec294484fe9ac61b2e718311b4c-f83faf506c3e49d4ade5f48881d9d1a7
I20240503 02:19:13.267840 29889 tablet_copy_source_session.cc:205] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:13.269497 31548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83faf506c3e49d4ade5f48881d9d1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:13.273761 31548 tablet_copy_client.cc:783] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:13.274003 31548 tablet_copy_client.cc:647] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:13.275950 31548 tablet_copy_client.cc:523] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:13.278131 31548 tablet_bootstrap.cc:492] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c: Bootstrap starting.
I20240503 02:19:13.307473 31548 tablet_bootstrap.cc:492] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:13.307878 31548 tablet_bootstrap.cc:492] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c: Bootstrap complete.
I20240503 02:19:13.308099 31548 ts_tablet_manager.cc:1367] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c: Time spent bootstrapping tablet: real 0.030s	user 0.026s	sys 0.004s
I20240503 02:19:13.308844 31548 raft_consensus.cc:348] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } }
I20240503 02:19:13.309082 31548 raft_consensus.cc:729] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 319ccec294484fe9ac61b2e718311b4c, State: Initialized, Role: LEARNER
I20240503 02:19:13.309285 31548 consensus_queue.cc:260] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } }
I20240503 02:19:13.309962 31548 ts_tablet_manager.cc:1398] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 02:19:13.310837 29889 tablet_copy_service.cc:342] P 6c36b0a7f8a3485f9a9d62e3e7793f25: Request end of tablet copy session 319ccec294484fe9ac61b2e718311b4c-f83faf506c3e49d4ade5f48881d9d1a7 received from {username='slave'} at 127.0.145.4:52013
I20240503 02:19:13.311034 29889 tablet_copy_service.cc:433] P 6c36b0a7f8a3485f9a9d62e3e7793f25: ending tablet copy session 319ccec294484fe9ac61b2e718311b4c-f83faf506c3e49d4ade5f48881d9d1a7 on tablet f83faf506c3e49d4ade5f48881d9d1a7 with peer 319ccec294484fe9ac61b2e718311b4c
I20240503 02:19:13.446184 31548 ts_tablet_manager.cc:902] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c: Initiating tablet copy from peer 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557)
I20240503 02:19:13.447376 31548 tablet_copy_client.cc:308] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:46557
I20240503 02:19:13.448065 29625 tablet_copy_service.cc:140] P 07bd09ad30cc4932bfff380344c8e9fc: Received BeginTabletCopySession request for tablet 2451cf2c781a44e0882f111cb29fa40f from peer 319ccec294484fe9ac61b2e718311b4c ({username='slave'} at 127.0.145.4:48669)
I20240503 02:19:13.448215 29625 tablet_copy_service.cc:161] P 07bd09ad30cc4932bfff380344c8e9fc: Beginning new tablet copy session on tablet 2451cf2c781a44e0882f111cb29fa40f from peer 319ccec294484fe9ac61b2e718311b4c at {username='slave'} at 127.0.145.4:48669: session id = 319ccec294484fe9ac61b2e718311b4c-2451cf2c781a44e0882f111cb29fa40f
I20240503 02:19:13.450055 29625 tablet_copy_source_session.cc:205] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:13.451267 31548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2451cf2c781a44e0882f111cb29fa40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:13.455633 31548 tablet_copy_client.cc:783] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:13.455835 31548 tablet_copy_client.cc:647] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:13.457337 31548 tablet_copy_client.cc:523] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:13.459640 31548 tablet_bootstrap.cc:492] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c: Bootstrap starting.
I20240503 02:19:13.479319 31576 ts_tablet_manager.cc:902] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd: Initiating tablet copy from peer 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133)
I20240503 02:19:13.480968 31576 tablet_copy_client.cc:308] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:41133
I20240503 02:19:13.487742 29889 tablet_copy_service.cc:140] P 6c36b0a7f8a3485f9a9d62e3e7793f25: Received BeginTabletCopySession request for tablet 899b3fc0c13540e485d0b76417c205e4 from peer 209bba2a25244371943e842bdf9267cd ({username='slave'} at 127.0.145.5:44607)
I20240503 02:19:13.487903 29889 tablet_copy_service.cc:161] P 6c36b0a7f8a3485f9a9d62e3e7793f25: Beginning new tablet copy session on tablet 899b3fc0c13540e485d0b76417c205e4 from peer 209bba2a25244371943e842bdf9267cd at {username='slave'} at 127.0.145.5:44607: session id = 209bba2a25244371943e842bdf9267cd-899b3fc0c13540e485d0b76417c205e4
I20240503 02:19:13.490413 29889 tablet_copy_source_session.cc:205] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:13.491668 31576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899b3fc0c13540e485d0b76417c205e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:13.495169 31548 tablet_bootstrap.cc:492] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:13.495545 31548 tablet_bootstrap.cc:492] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c: Bootstrap complete.
I20240503 02:19:13.495795 31548 ts_tablet_manager.cc:1367] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c: Time spent bootstrapping tablet: real 0.036s	user 0.028s	sys 0.008s
I20240503 02:19:13.496095 31576 tablet_copy_client.cc:783] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:13.496330 31576 tablet_copy_client.cc:647] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:13.496527 31548 raft_consensus.cc:348] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } }
I20240503 02:19:13.496811 31548 raft_consensus.cc:729] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 319ccec294484fe9ac61b2e718311b4c, State: Initialized, Role: LEARNER
I20240503 02:19:13.497045 31548 consensus_queue.cc:260] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } }
I20240503 02:19:13.497709 31548 ts_tablet_manager.cc:1398] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 02:19:13.498416 29625 tablet_copy_service.cc:342] P 07bd09ad30cc4932bfff380344c8e9fc: Request end of tablet copy session 319ccec294484fe9ac61b2e718311b4c-2451cf2c781a44e0882f111cb29fa40f received from {username='slave'} at 127.0.145.4:48669
I20240503 02:19:13.498572 29625 tablet_copy_service.cc:433] P 07bd09ad30cc4932bfff380344c8e9fc: ending tablet copy session 319ccec294484fe9ac61b2e718311b4c-2451cf2c781a44e0882f111cb29fa40f on tablet 2451cf2c781a44e0882f111cb29fa40f with peer 319ccec294484fe9ac61b2e718311b4c
I20240503 02:19:13.498736 31576 tablet_copy_client.cc:523] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:13.501721 31576 tablet_bootstrap.cc:492] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd: Bootstrap starting.
I20240503 02:19:13.529816 31576 tablet_bootstrap.cc:492] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:13.530210 31576 tablet_bootstrap.cc:492] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd: Bootstrap complete.
I20240503 02:19:13.530453 31576 ts_tablet_manager.cc:1367] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd: Time spent bootstrapping tablet: real 0.029s	user 0.027s	sys 0.000s
I20240503 02:19:13.531108 31576 raft_consensus.cc:348] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } }
I20240503 02:19:13.531348 31576 raft_consensus.cc:729] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 209bba2a25244371943e842bdf9267cd, State: Initialized, Role: LEARNER
I20240503 02:19:13.531571 31576 consensus_queue.cc:260] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: true } }
I20240503 02:19:13.532392 31576 ts_tablet_manager.cc:1398] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd: Time spent starting tablet: real 0.002s	user 0.006s	sys 0.000s
I20240503 02:19:13.533301 29889 tablet_copy_service.cc:342] P 6c36b0a7f8a3485f9a9d62e3e7793f25: Request end of tablet copy session 209bba2a25244371943e842bdf9267cd-899b3fc0c13540e485d0b76417c205e4 received from {username='slave'} at 127.0.145.5:44607
I20240503 02:19:13.533429 29889 tablet_copy_service.cc:433] P 6c36b0a7f8a3485f9a9d62e3e7793f25: ending tablet copy session 209bba2a25244371943e842bdf9267cd-899b3fc0c13540e485d0b76417c205e4 on tablet 899b3fc0c13540e485d0b76417c205e4 with peer 209bba2a25244371943e842bdf9267cd
I20240503 02:19:13.769107 30791 raft_consensus.cc:1203] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:13.906837 30791 raft_consensus.cc:1203] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:13.953891 30921 raft_consensus.cc:1203] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:14.214274 31600 raft_consensus.cc:1050] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25: attempting to promote NON_VOTER 319ccec294484fe9ac61b2e718311b4c to VOTER
I20240503 02:19:14.215170 31600 consensus_queue.cc:237] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } }
I20240503 02:19:14.217577 29736 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 6c36b0a7f8a3485f9a9d62e3e7793f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:14.217576 30791 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Refusing update from remote peer 6c36b0a7f8a3485f9a9d62e3e7793f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:14.217602 29606 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Refusing update from remote peer 6c36b0a7f8a3485f9a9d62e3e7793f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:14.218122 31600 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:14.218446 31599 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:14.218793 31634 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:14.222406 31600 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:14.223055 29736 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:14.223079 30791 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:14.228515 29461 catalog_manager.cc:5484] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 reported cstate change: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:14.228638 29606 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:14.341632 29869 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f83faf506c3e49d4ade5f48881d9d1a7"
dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34766
I20240503 02:19:14.341869 29869 raft_consensus.cc:595] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:19:14.367264 31634 raft_consensus.cc:1050] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25: attempting to promote NON_VOTER 209bba2a25244371943e842bdf9267cd to VOTER
I20240503 02:19:14.368003 31634 consensus_queue.cc:237] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } }
I20240503 02:19:14.370081 29736 raft_consensus.cc:1261] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 6c36b0a7f8a3485f9a9d62e3e7793f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:14.370107 30921 raft_consensus.cc:1261] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd [term 1 LEARNER]: Refusing update from remote peer 6c36b0a7f8a3485f9a9d62e3e7793f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:14.370162 29606 raft_consensus.cc:1261] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Refusing update from remote peer 6c36b0a7f8a3485f9a9d62e3e7793f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:14.370682 31634 consensus_queue.cc:1035] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:14.371001 30549 consensus_queue.cc:1035] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:14.371239 31599 consensus_queue.cc:1035] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:14.374502 31599 raft_consensus.cc:2932] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 209bba2a25244371943e842bdf9267cd (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:14.375192 29736 raft_consensus.cc:2932] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 209bba2a25244371943e842bdf9267cd (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:14.375327 29606 raft_consensus.cc:2932] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 209bba2a25244371943e842bdf9267cd (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:14.378157 30921 raft_consensus.cc:2932] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 209bba2a25244371943e842bdf9267cd (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:14.379082 29457 catalog_manager.cc:5484] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 reported cstate change: config changed from index 4 to 5, 209bba2a25244371943e842bdf9267cd (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:14.384212 29869 consensus_queue.cc:237] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } }
I20240503 02:19:14.385998 30921 raft_consensus.cc:1261] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd [term 1 FOLLOWER]: Refusing update from remote peer 6c36b0a7f8a3485f9a9d62e3e7793f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:14.386351 29736 raft_consensus.cc:1261] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 6c36b0a7f8a3485f9a9d62e3e7793f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:14.386718 31599 consensus_queue.cc:1035] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:14.387153 31634 consensus_queue.cc:1035] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:14.389026 31634 raft_consensus.cc:2932] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:14.389806 30921 raft_consensus.cc:2932] T 899b3fc0c13540e485d0b76417c205e4 P 209bba2a25244371943e842bdf9267cd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:14.390336 29736 raft_consensus.cc:2932] T 899b3fc0c13540e485d0b76417c205e4 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } } }
I20240503 02:19:14.392030 29451 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 899b3fc0c13540e485d0b76417c205e4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:14.393097 29460 catalog_manager.cc:5484] T 899b3fc0c13540e485d0b76417c205e4 P 6c36b0a7f8a3485f9a9d62e3e7793f25 reported cstate change: config changed from index 5 to 6, VOTER 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "209bba2a25244371943e842bdf9267cd" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 35793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:14.396735 29586 tablet_service.cc:1514] Processing DeleteTablet for tablet 899b3fc0c13540e485d0b76417c205e4 with delete_type TABLET_DATA_TOMBSTONED (TS 07bd09ad30cc4932bfff380344c8e9fc not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48014
I20240503 02:19:14.399013 31666 tablet_replica.cc:330] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc: stopping tablet replica
I20240503 02:19:14.399219 31666 raft_consensus.cc:2229] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:14.399510 31666 raft_consensus.cc:2258] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:14.400276 31666 ts_tablet_manager.cc:1875] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:14.404826 31666 ts_tablet_manager.cc:1888] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:19:14.404942 31666 log.cc:1196] T 899b3fc0c13540e485d0b76417c205e4 P 07bd09ad30cc4932bfff380344c8e9fc: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/899b3fc0c13540e485d0b76417c205e4
I20240503 02:19:14.405723 29450 catalog_manager.cc:4830] TS 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557): tablet 899b3fc0c13540e485d0b76417c205e4 (table rebalance_test_table_2 [id=a25280dd90f64d31a487b2c771383e8a]) successfully deleted
I20240503 02:19:14.439330 31616 raft_consensus.cc:1050] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc: attempting to promote NON_VOTER 319ccec294484fe9ac61b2e718311b4c to VOTER
I20240503 02:19:14.440013 31616 consensus_queue.cc:237] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } }
I20240503 02:19:14.443001 30791 raft_consensus.cc:1261] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:14.443045 29869 raft_consensus.cc:1261] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:14.443567 29736 raft_consensus.cc:1261] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:14.443671 31617 consensus_queue.cc:1035] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:14.444041 31668 consensus_queue.cc:1035] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:14.444381 31616 consensus_queue.cc:1035] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:14.447784 31616 raft_consensus.cc:2932] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:14.448370 29736 raft_consensus.cc:2932] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:14.452545 29869 raft_consensus.cc:2932] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:14.453361 29457 catalog_manager.cc:5484] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba reported cstate change: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:14.455762 30791 raft_consensus.cc:2932] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:14.460808 29606 consensus_queue.cc:237] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } }
I20240503 02:19:14.462556 30791 raft_consensus.cc:1261] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:14.462586 29736 raft_consensus.cc:1261] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:14.463109 31668 consensus_queue.cc:1035] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:14.463390 31617 consensus_queue.cc:1035] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:14.465457 31616 raft_consensus.cc:2932] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:14.468248 29450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2451cf2c781a44e0882f111cb29fa40f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:14.468607 30791 raft_consensus.cc:2932] T 2451cf2c781a44e0882f111cb29fa40f P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:14.468662 29736 raft_consensus.cc:2932] T 2451cf2c781a44e0882f111cb29fa40f P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:14.469302 29460 catalog_manager.cc:5484] T 2451cf2c781a44e0882f111cb29fa40f P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: config changed from index 5 to 6, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:14.472505 29849 tablet_service.cc:1514] Processing DeleteTablet for tablet 2451cf2c781a44e0882f111cb29fa40f with delete_type TABLET_DATA_TOMBSTONED (TS 6c36b0a7f8a3485f9a9d62e3e7793f25 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38302
I20240503 02:19:14.474002 31672 tablet_replica.cc:330] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25: stopping tablet replica
I20240503 02:19:14.474252 31672 raft_consensus.cc:2229] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:14.474686 31672 raft_consensus.cc:2258] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:14.475628 31672 ts_tablet_manager.cc:1875] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:14.483070 31672 ts_tablet_manager.cc:1888] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:19:14.483191 31672 log.cc:1196] T 2451cf2c781a44e0882f111cb29fa40f P 6c36b0a7f8a3485f9a9d62e3e7793f25: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/2451cf2c781a44e0882f111cb29fa40f
I20240503 02:19:14.483968 29451 catalog_manager.cc:4830] TS 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133): tablet 2451cf2c781a44e0882f111cb29fa40f (table rebalance_test_table_1 [id=11a9093ed71046c1b45b6520ebcc4ed0]) successfully deleted
I20240503 02:19:14.578186 29869 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f83faf506c3e49d4ade5f48881d9d1a7"
dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34788
I20240503 02:19:14.578384 29869 raft_consensus.cc:595] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:19:14.595715 29869 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f83faf506c3e49d4ade5f48881d9d1a7"
dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34794
I20240503 02:19:14.595916 29869 raft_consensus.cc:595] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:19:14.612799 30549 raft_consensus.cc:979] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25: : Instructing follower 07bd09ad30cc4932bfff380344c8e9fc to start an election
I20240503 02:19:14.612969 30549 raft_consensus.cc:1067] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEADER]: Signalling peer 07bd09ad30cc4932bfff380344c8e9fc to start an election
I20240503 02:19:14.613745 29606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f83faf506c3e49d4ade5f48881d9d1a7"
dest_uuid: "07bd09ad30cc4932bfff380344c8e9fc"
 from {username='slave'} at 127.0.145.3:35531
I20240503 02:19:14.614003 29606 raft_consensus.cc:482] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:19:14.614113 29606 raft_consensus.cc:3037] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:19:14.616209 29606 raft_consensus.cc:504] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } }
I20240503 02:19:14.617074 29606 leader_election.cc:288] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 2 election: Requested vote from peers 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371), 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133), 319ccec294484fe9ac61b2e718311b4c (127.0.145.4:39727)
I20240503 02:19:14.617484 29736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83faf506c3e49d4ade5f48881d9d1a7" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "99f937fbfb7c49f2afe278eaa73edbba"
I20240503 02:19:14.617611 29869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83faf506c3e49d4ade5f48881d9d1a7" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25"
I20240503 02:19:14.617720 29736 raft_consensus.cc:3037] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:19:14.617825 29869 raft_consensus.cc:3032] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:19:14.617949 29869 raft_consensus.cc:729] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c36b0a7f8a3485f9a9d62e3e7793f25, State: Running, Role: LEADER
I20240503 02:19:14.617867 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f83faf506c3e49d4ade5f48881d9d1a7" candidate_uuid: "07bd09ad30cc4932bfff380344c8e9fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "319ccec294484fe9ac61b2e718311b4c"
I20240503 02:19:14.618254 30791 raft_consensus.cc:3037] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:19:14.618381 29869 consensus_queue.cc:260] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } }
I20240503 02:19:14.618956 29869 raft_consensus.cc:3037] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:19:14.620672 29736 raft_consensus.cc:2454] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 2.
I20240503 02:19:14.620994 29869 raft_consensus.cc:2454] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 2.
I20240503 02:19:14.621289 30791 raft_consensus.cc:2454] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd09ad30cc4932bfff380344c8e9fc in term 2.
I20240503 02:19:14.621491 29541 leader_election.cc:302] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 07bd09ad30cc4932bfff380344c8e9fc, 6c36b0a7f8a3485f9a9d62e3e7793f25, 99f937fbfb7c49f2afe278eaa73edbba; no voters: 
I20240503 02:19:14.621834 31616 raft_consensus.cc:2781] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:19:14.622009 31616 raft_consensus.cc:686] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 2 LEADER]: Becoming Leader. State: Replica: 07bd09ad30cc4932bfff380344c8e9fc, State: Running, Role: LEADER
I20240503 02:19:14.622251 31616 consensus_queue.cc:237] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } }
I20240503 02:19:14.624967 29457 catalog_manager.cc:5484] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: term changed from 1 to 2, leader changed from 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) to 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1). New cstate: current_term: 2 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:15.020090 29869 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 2 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:19:15.020732 31617 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:15.035282 30791 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 2 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:19:15.035876 31617 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:15.040638 29736 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 2 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:19:15.041244 31616 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:15.507155 29606 consensus_queue.cc:237] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } }
I20240503 02:19:15.509006 30791 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 2 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:19:15.509088 29736 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 2 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:19:15.509584 31617 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:19:15.509894 31717 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:19:15.512337 31717 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:15.512914 29736 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:15.515213 29450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f83faf506c3e49d4ade5f48881d9d1a7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:15.515668 30791 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:15.517298 29460 catalog_manager.cc:5484] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba reported cstate change: config changed from index 5 to 7, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New cstate: current_term: 2 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:15.521821 29849 tablet_service.cc:1514] Processing DeleteTablet for tablet f83faf506c3e49d4ade5f48881d9d1a7 with delete_type TABLET_DATA_TOMBSTONED (TS 6c36b0a7f8a3485f9a9d62e3e7793f25 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38302
I20240503 02:19:15.522432 31733 tablet_replica.cc:330] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25: stopping tablet replica
I20240503 02:19:15.522680 31733 raft_consensus.cc:2229] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:15.522935 31733 raft_consensus.cc:2258] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:15.523842 31733 ts_tablet_manager.cc:1875] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:15.528249 31733 ts_tablet_manager.cc:1888] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:19:15.528368 31733 log.cc:1196] T f83faf506c3e49d4ade5f48881d9d1a7 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/f83faf506c3e49d4ade5f48881d9d1a7
I20240503 02:19:15.529124 29451 catalog_manager.cc:4830] TS 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133): tablet f83faf506c3e49d4ade5f48881d9d1a7 (table rebalance_test_table_2 [id=a25280dd90f64d31a487b2c771383e8a]) successfully deleted
I20240503 02:19:15.739122 29586 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:15.761512 30901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:15.770460 30771 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:15.774073 29849 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:15.799387 30640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:15.806941 29716 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:15.959388 29586 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:15.970304 30901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:15.983143 29716 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:15.984099 30771 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:15.986424 29849 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:15.993034 30640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:16.079562 29606 consensus_queue.cc:237] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: true } }
I20240503 02:19:16.082772 29736 raft_consensus.cc:1261] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:19:16.083101 30791 raft_consensus.cc:1261] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:19:16.083703 31617 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:19:16.084189 31717 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:19:16.085206 29541 consensus_peers.cc:485] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc -> Peer 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133): Couldn't send request to peer 6c36b0a7f8a3485f9a9d62e3e7793f25. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:16.087378 29606 consensus_queue.cc:237] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } }
I20240503 02:19:16.088196 31617 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: true } } }
I20240503 02:19:16.088927 29736 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: true } } }
I20240503 02:19:16.091148 30790 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 2 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:19:16.092200 31824 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:16.093129 29738 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 2 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:19:16.094834 31717 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:19:16.095132 29541 consensus_peers.cc:485] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc -> Peer 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6:39629): Couldn't send request to peer 58a2474aadf746cc9d50a38f1ee9f80c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f83faf506c3e49d4ade5f48881d9d1a7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:16.095727 31717 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
I20240503 02:19:16.096249 30791 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: true } } }
I20240503 02:19:16.097040 29457 catalog_manager.cc:5484] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: config changed from index 6 to 7, NON_VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) added. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:16.099361 30790 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
I20240503 02:19:16.103369 29738 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
I20240503 02:19:16.123524 29460 catalog_manager.cc:5484] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c reported cstate change: config changed from index 7 to 8, NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) added. New cstate: current_term: 2 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } } }
I20240503 02:19:16.140115 29605 consensus_queue.cc:237] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: true } }
I20240503 02:19:16.142903 29869 raft_consensus.cc:1261] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:19:16.143116 29539 consensus_peers.cc:485] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc -> Peer 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371): Couldn't send request to peer 99f937fbfb7c49f2afe278eaa73edbba. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:16.143254 30660 raft_consensus.cc:1261] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:19:16.143514 31827 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:19:16.143838 31824 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:19:16.146728 31824 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: true } } }
I20240503 02:19:16.147450 29869 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: true } } }
I20240503 02:19:16.152408 29464 catalog_manager.cc:5484] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 reported cstate change: config changed from index 6 to 7, NON_VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) added. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: true } } }
I20240503 02:19:16.153200 30660 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: true } } }
I20240503 02:19:16.257865 29586 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:16.261238 30901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:16.264415 30640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:16.274498 30771 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:16.294129 29849 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:16.298341 29716 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:16.363014 29606 consensus_queue.cc:237] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } }
I20240503 02:19:16.365293 30660 raft_consensus.cc:1261] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:19:16.365355 29541 consensus_peers.cc:485] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc -> Peer 319ccec294484fe9ac61b2e718311b4c (127.0.145.4:39727): Couldn't send request to peer 319ccec294484fe9ac61b2e718311b4c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 548b1357845a405aad60446d2c5d120c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:16.365377 29869 raft_consensus.cc:1261] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:19:16.365828 31617 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:19:16.366454 31827 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:19:16.369119 31617 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
I20240503 02:19:16.371245 29869 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
I20240503 02:19:16.371680 30660 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } }
I20240503 02:19:16.373768 29464 catalog_manager.cc:5484] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: config changed from index 6 to 7, NON_VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) added. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:16.470800 31913 ts_tablet_manager.cc:902] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Initiating tablet copy from peer 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557)
I20240503 02:19:16.472441 31913 tablet_copy_client.cc:308] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:46557
I20240503 02:19:16.473336 29625 tablet_copy_service.cc:140] P 07bd09ad30cc4932bfff380344c8e9fc: Received BeginTabletCopySession request for tablet 6f4fdecce7bd47b9a92d66a775211292 from peer 6c36b0a7f8a3485f9a9d62e3e7793f25 ({username='slave'} at 127.0.145.3:35531)
I20240503 02:19:16.473492 29625 tablet_copy_service.cc:161] P 07bd09ad30cc4932bfff380344c8e9fc: Beginning new tablet copy session on tablet 6f4fdecce7bd47b9a92d66a775211292 from peer 6c36b0a7f8a3485f9a9d62e3e7793f25 at {username='slave'} at 127.0.145.3:35531: session id = 6c36b0a7f8a3485f9a9d62e3e7793f25-6f4fdecce7bd47b9a92d66a775211292
I20240503 02:19:16.475445 29625 tablet_copy_source_session.cc:205] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:16.476926 31913 ts_tablet_manager.cc:1875] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:19:16.483698 31913 ts_tablet_manager.cc:1888] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:19:16.484020 31913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4fdecce7bd47b9a92d66a775211292. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:16.487383 31913 tablet_copy_client.cc:783] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:16.487696 31913 tablet_copy_client.cc:647] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:16.489866 31913 tablet_copy_client.cc:523] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:16.492957 31913 tablet_bootstrap.cc:492] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Bootstrap starting.
I20240503 02:19:16.524873 31913 tablet_bootstrap.cc:492] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:16.525308 31913 tablet_bootstrap.cc:492] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Bootstrap complete.
I20240503 02:19:16.525553 31913 ts_tablet_manager.cc:1367] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent bootstrapping tablet: real 0.033s	user 0.028s	sys 0.004s
I20240503 02:19:16.526151 31913 raft_consensus.cc:348] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: true } }
I20240503 02:19:16.526373 31913 raft_consensus.cc:729] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c36b0a7f8a3485f9a9d62e3e7793f25, State: Initialized, Role: LEARNER
I20240503 02:19:16.526569 31913 consensus_queue.cc:260] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: true } }
I20240503 02:19:16.527103 31913 ts_tablet_manager.cc:1398] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240503 02:19:16.527966 29625 tablet_copy_service.cc:342] P 07bd09ad30cc4932bfff380344c8e9fc: Request end of tablet copy session 6c36b0a7f8a3485f9a9d62e3e7793f25-6f4fdecce7bd47b9a92d66a775211292 received from {username='slave'} at 127.0.145.3:35531
I20240503 02:19:16.528077 29625 tablet_copy_service.cc:433] P 07bd09ad30cc4932bfff380344c8e9fc: ending tablet copy session 6c36b0a7f8a3485f9a9d62e3e7793f25-6f4fdecce7bd47b9a92d66a775211292 on tablet 6f4fdecce7bd47b9a92d66a775211292 with peer 6c36b0a7f8a3485f9a9d62e3e7793f25
I20240503 02:19:16.546358 31919 ts_tablet_manager.cc:902] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c: Initiating tablet copy from peer 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557)
I20240503 02:19:16.547891 31919 tablet_copy_client.cc:308] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:46557
I20240503 02:19:16.548626 29625 tablet_copy_service.cc:140] P 07bd09ad30cc4932bfff380344c8e9fc: Received BeginTabletCopySession request for tablet f83faf506c3e49d4ade5f48881d9d1a7 from peer 58a2474aadf746cc9d50a38f1ee9f80c ({username='slave'} at 127.0.145.6:45531)
I20240503 02:19:16.548782 29625 tablet_copy_service.cc:161] P 07bd09ad30cc4932bfff380344c8e9fc: Beginning new tablet copy session on tablet f83faf506c3e49d4ade5f48881d9d1a7 from peer 58a2474aadf746cc9d50a38f1ee9f80c at {username='slave'} at 127.0.145.6:45531: session id = 58a2474aadf746cc9d50a38f1ee9f80c-f83faf506c3e49d4ade5f48881d9d1a7
I20240503 02:19:16.550731 29625 tablet_copy_source_session.cc:205] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:16.552029 31919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f83faf506c3e49d4ade5f48881d9d1a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:16.557271 31919 tablet_copy_client.cc:783] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:16.557586 31919 tablet_copy_client.cc:647] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:16.559422 31919 tablet_copy_client.cc:523] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:16.562084 31919 tablet_bootstrap.cc:492] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c: Bootstrap starting.
I20240503 02:19:16.595139 31919 tablet_bootstrap.cc:492] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:16.595690 31919 tablet_bootstrap.cc:492] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c: Bootstrap complete.
I20240503 02:19:16.596007 31919 ts_tablet_manager.cc:1367] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c: Time spent bootstrapping tablet: real 0.034s	user 0.031s	sys 0.004s
I20240503 02:19:16.596943 31919 raft_consensus.cc:348] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } }
I20240503 02:19:16.597283 31919 raft_consensus.cc:729] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 58a2474aadf746cc9d50a38f1ee9f80c, State: Initialized, Role: LEARNER
I20240503 02:19:16.597505 31919 consensus_queue.cc:260] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: NON_VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: true } }
I20240503 02:19:16.598083 31919 ts_tablet_manager.cc:1398] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:19:16.598886 29625 tablet_copy_service.cc:342] P 07bd09ad30cc4932bfff380344c8e9fc: Request end of tablet copy session 58a2474aadf746cc9d50a38f1ee9f80c-f83faf506c3e49d4ade5f48881d9d1a7 received from {username='slave'} at 127.0.145.6:45531
I20240503 02:19:16.599045 29625 tablet_copy_service.cc:433] P 07bd09ad30cc4932bfff380344c8e9fc: ending tablet copy session 58a2474aadf746cc9d50a38f1ee9f80c-f83faf506c3e49d4ade5f48881d9d1a7 on tablet f83faf506c3e49d4ade5f48881d9d1a7 with peer 58a2474aadf746cc9d50a38f1ee9f80c
I20240503 02:19:16.668458 31945 ts_tablet_manager.cc:902] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: Initiating tablet copy from peer 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557)
I20240503 02:19:16.670044 31945 tablet_copy_client.cc:308] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:46557
I20240503 02:19:16.670877 29625 tablet_copy_service.cc:140] P 07bd09ad30cc4932bfff380344c8e9fc: Received BeginTabletCopySession request for tablet 9e13aebe3d3649dda2b2a4f157c08dbc from peer 99f937fbfb7c49f2afe278eaa73edbba ({username='slave'} at 127.0.145.2:59787)
I20240503 02:19:16.671042 29625 tablet_copy_service.cc:161] P 07bd09ad30cc4932bfff380344c8e9fc: Beginning new tablet copy session on tablet 9e13aebe3d3649dda2b2a4f157c08dbc from peer 99f937fbfb7c49f2afe278eaa73edbba at {username='slave'} at 127.0.145.2:59787: session id = 99f937fbfb7c49f2afe278eaa73edbba-9e13aebe3d3649dda2b2a4f157c08dbc
I20240503 02:19:16.672942 29625 tablet_copy_source_session.cc:205] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:16.674189 31945 ts_tablet_manager.cc:1875] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:19:16.678972 31945 ts_tablet_manager.cc:1888] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:19:16.679277 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e13aebe3d3649dda2b2a4f157c08dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:16.682518 31945 tablet_copy_client.cc:783] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:16.682902 31945 tablet_copy_client.cc:647] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:16.685050 31945 tablet_copy_client.cc:523] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:16.688042 31945 tablet_bootstrap.cc:492] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: Bootstrap starting.
I20240503 02:19:16.724057 31945 tablet_bootstrap.cc:492] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:16.724550 31945 tablet_bootstrap.cc:492] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: Bootstrap complete.
I20240503 02:19:16.724826 31945 ts_tablet_manager.cc:1367] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: Time spent bootstrapping tablet: real 0.037s	user 0.032s	sys 0.004s
I20240503 02:19:16.725440 31945 raft_consensus.cc:348] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: true } }
I20240503 02:19:16.725653 31945 raft_consensus.cc:729] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99f937fbfb7c49f2afe278eaa73edbba, State: Initialized, Role: LEARNER
I20240503 02:19:16.725811 31945 consensus_queue.cc:260] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: true } }
I20240503 02:19:16.726308 31945 ts_tablet_manager.cc:1398] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:19:16.727142 29625 tablet_copy_service.cc:342] P 07bd09ad30cc4932bfff380344c8e9fc: Request end of tablet copy session 99f937fbfb7c49f2afe278eaa73edbba-9e13aebe3d3649dda2b2a4f157c08dbc received from {username='slave'} at 127.0.145.2:59787
I20240503 02:19:16.727254 29625 tablet_copy_service.cc:433] P 07bd09ad30cc4932bfff380344c8e9fc: ending tablet copy session 99f937fbfb7c49f2afe278eaa73edbba-9e13aebe3d3649dda2b2a4f157c08dbc on tablet 9e13aebe3d3649dda2b2a4f157c08dbc with peer 99f937fbfb7c49f2afe278eaa73edbba
I20240503 02:19:16.967782 31971 ts_tablet_manager.cc:902] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c: Initiating tablet copy from peer 07bd09ad30cc4932bfff380344c8e9fc (127.0.145.1:46557)
I20240503 02:19:16.969355 31971 tablet_copy_client.cc:308] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:46557
I20240503 02:19:16.970104 29625 tablet_copy_service.cc:140] P 07bd09ad30cc4932bfff380344c8e9fc: Received BeginTabletCopySession request for tablet 548b1357845a405aad60446d2c5d120c from peer 319ccec294484fe9ac61b2e718311b4c ({username='slave'} at 127.0.145.4:48669)
I20240503 02:19:16.970278 29625 tablet_copy_service.cc:161] P 07bd09ad30cc4932bfff380344c8e9fc: Beginning new tablet copy session on tablet 548b1357845a405aad60446d2c5d120c from peer 319ccec294484fe9ac61b2e718311b4c at {username='slave'} at 127.0.145.4:48669: session id = 319ccec294484fe9ac61b2e718311b4c-548b1357845a405aad60446d2c5d120c
I20240503 02:19:16.972241 29625 tablet_copy_source_session.cc:205] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:16.973560 31971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 548b1357845a405aad60446d2c5d120c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:16.977810 31971 tablet_copy_client.cc:783] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:16.978174 31971 tablet_copy_client.cc:647] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:16.979888 31971 tablet_copy_client.cc:523] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:16.982122 31971 tablet_bootstrap.cc:492] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c: Bootstrap starting.
I20240503 02:19:17.005494 29869 raft_consensus.cc:1203] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:19:17.014118 31971 tablet_bootstrap.cc:492] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:17.014592 31971 tablet_bootstrap.cc:492] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c: Bootstrap complete.
I20240503 02:19:17.014916 31971 ts_tablet_manager.cc:1367] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c: Time spent bootstrapping tablet: real 0.033s	user 0.031s	sys 0.003s
I20240503 02:19:17.015615 31971 raft_consensus.cc:348] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } }
I20240503 02:19:17.015830 31971 raft_consensus.cc:729] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 319ccec294484fe9ac61b2e718311b4c, State: Initialized, Role: LEARNER
I20240503 02:19:17.016054 31971 consensus_queue.cc:260] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } }
I20240503 02:19:17.016886 31971 ts_tablet_manager.cc:1398] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:19:17.017663 29625 tablet_copy_service.cc:342] P 07bd09ad30cc4932bfff380344c8e9fc: Request end of tablet copy session 319ccec294484fe9ac61b2e718311b4c-548b1357845a405aad60446d2c5d120c received from {username='slave'} at 127.0.145.4:48669
I20240503 02:19:17.017853 29625 tablet_copy_service.cc:433] P 07bd09ad30cc4932bfff380344c8e9fc: ending tablet copy session 319ccec294484fe9ac61b2e718311b4c-548b1357845a405aad60446d2c5d120c on tablet 548b1357845a405aad60446d2c5d120c with peer 319ccec294484fe9ac61b2e718311b4c
I20240503 02:19:17.045620 29736 raft_consensus.cc:1203] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:19:17.098253 30660 raft_consensus.cc:1203] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240503 02:19:17.345539 30790 raft_consensus.cc:1203] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:19:17.446305 31978 raft_consensus.cc:1050] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc: attempting to promote NON_VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 to VOTER
I20240503 02:19:17.447183 31978 consensus_queue.cc:237] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: false } }
I20240503 02:19:17.449458 29869 raft_consensus.cc:1261] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 LEARNER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:17.449666 30790 raft_consensus.cc:1261] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:17.449857 29736 raft_consensus.cc:1261] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:17.449949 31978 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:17.450271 31617 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:17.450634 31977 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:17.454018 31978 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: false } } }
I20240503 02:19:17.454660 29736 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: false } } }
I20240503 02:19:17.454645 29869 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: false } } }
I20240503 02:19:17.457711 30790 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: false } } }
I20240503 02:19:17.461623 29460 catalog_manager.cc:5484] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: config changed from index 7 to 8, 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:17.472726 29606 consensus_queue.cc:237] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: false } }
I20240503 02:19:17.474398 30790 raft_consensus.cc:1261] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:19:17.474495 29869 raft_consensus.cc:1261] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:19:17.474987 31617 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:19:17.475287 31977 consensus_queue.cc:1035] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:19:17.477463 31617 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: false } } }
I20240503 02:19:17.480952 30790 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: false } } }
I20240503 02:19:17.481783 29450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6f4fdecce7bd47b9a92d66a775211292 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:17.482451 29464 catalog_manager.cc:5484] T 6f4fdecce7bd47b9a92d66a775211292 P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: config changed from index 8 to 9, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:17.483767 29869 raft_consensus.cc:2932] T 6f4fdecce7bd47b9a92d66a775211292 P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { promote: false } } }
I20240503 02:19:17.489233 29716 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f4fdecce7bd47b9a92d66a775211292 with delete_type TABLET_DATA_TOMBSTONED (TS 99f937fbfb7c49f2afe278eaa73edbba not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48100
I20240503 02:19:17.489935 32031 tablet_replica.cc:330] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba: stopping tablet replica
I20240503 02:19:17.490239 32031 raft_consensus.cc:2229] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:17.490825 32031 raft_consensus.cc:2258] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:17.491878 32031 ts_tablet_manager.cc:1875] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:17.498059 32031 ts_tablet_manager.cc:1888] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:19:17.498167 32031 log.cc:1196] T 6f4fdecce7bd47b9a92d66a775211292 P 99f937fbfb7c49f2afe278eaa73edbba: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f4fdecce7bd47b9a92d66a775211292
I20240503 02:19:17.498907 29449 catalog_manager.cc:4830] TS 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2:37371): tablet 6f4fdecce7bd47b9a92d66a775211292 (table rebalance_test_table_2 [id=a25280dd90f64d31a487b2c771383e8a]) successfully deleted
I20240503 02:19:17.557477 31617 raft_consensus.cc:1050] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc: attempting to promote NON_VOTER 99f937fbfb7c49f2afe278eaa73edbba to VOTER
I20240503 02:19:17.558269 31617 consensus_queue.cc:237] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: false } }
I20240503 02:19:17.560461 30660 raft_consensus.cc:1261] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:17.560608 29736 raft_consensus.cc:1261] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 1 LEARNER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:17.561247 31978 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:17.561404 29869 raft_consensus.cc:1261] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:17.561833 31977 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:17.562094 31978 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:17.565299 31977 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: false } } }
I20240503 02:19:17.566326 30660 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: false } } }
I20240503 02:19:17.570178 29736 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: false } } }
I20240503 02:19:17.573911 29464 catalog_manager.cc:5484] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: config changed from index 7 to 8, 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:17.576316 31978 raft_consensus.cc:1050] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc: attempting to promote NON_VOTER 58a2474aadf746cc9d50a38f1ee9f80c to VOTER
I20240503 02:19:17.577067 31978 consensus_queue.cc:237] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } }
I20240503 02:19:17.580215 29869 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 99f937fbfb7c49f2afe278eaa73edbba (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } attrs { replace: true } } peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: false } } }
I20240503 02:19:17.583196 29606 consensus_queue.cc:237] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: false } }
I20240503 02:19:17.584493 30790 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 2 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:19:17.585723 29736 raft_consensus.cc:1261] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:19:17.585716 31978 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:19:17.586278 29738 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 2 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:19:17.586259 30659 raft_consensus.cc:1261] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:19:17.586330 31978 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:19:17.587384 30660 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c [term 2 LEARNER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:19:17.587934 31978 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:19:17.588171 31977 consensus_queue.cc:1035] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:19:17.588366 31978 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:19:17.591303 31977 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: false } } }
I20240503 02:19:17.592312 29738 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: false } } }
I20240503 02:19:17.592980 31617 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:17.594820 29736 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:17.595321 30660 raft_consensus.cc:2932] T 9e13aebe3d3649dda2b2a4f157c08dbc P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: false } } }
I20240503 02:19:17.597321 30659 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:17.598060 29461 catalog_manager.cc:5484] T 9e13aebe3d3649dda2b2a4f157c08dbc P 99f937fbfb7c49f2afe278eaa73edbba reported cstate change: config changed from index 8 to 9, VOTER 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } attrs { promote: false } } }
I20240503 02:19:17.598228 30790 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:17.601622 29464 catalog_manager.cc:5484] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c reported cstate change: config changed from index 8 to 9, 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false replace: true } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:17.603080 29849 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e13aebe3d3649dda2b2a4f157c08dbc with delete_type TABLET_DATA_TOMBSTONED (TS 6c36b0a7f8a3485f9a9d62e3e7793f25 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38302
I20240503 02:19:17.604395 32066 tablet_replica.cc:330] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25: stopping tablet replica
I20240503 02:19:17.604606 32066 raft_consensus.cc:2229] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:17.605248 32066 raft_consensus.cc:2258] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:17.606431 29450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e13aebe3d3649dda2b2a4f157c08dbc with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:17.607530 29606 consensus_queue.cc:237] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } }
I20240503 02:19:17.608172 32066 ts_tablet_manager.cc:1875] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:17.609534 30660 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c [term 2 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:19:17.609611 29736 raft_consensus.cc:1261] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 2 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:19:17.610162 31617 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:19:17.611851 31617 consensus_queue.cc:1035] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:19:17.612622 32062 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 07bd09ad30cc4932bfff380344c8e9fc [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:17.613194 29738 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:17.615063 30659 raft_consensus.cc:2932] T f83faf506c3e49d4ade5f48881d9d1a7 P 58a2474aadf746cc9d50a38f1ee9f80c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:17.618019 29461 catalog_manager.cc:5484] T f83faf506c3e49d4ade5f48881d9d1a7 P 99f937fbfb7c49f2afe278eaa73edbba reported cstate change: config changed from index 9 to 10, VOTER 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) evicted. New cstate: current_term: 2 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "99f937fbfb7c49f2afe278eaa73edbba" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 37371 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false } } }
I20240503 02:19:17.618485 29450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f83faf506c3e49d4ade5f48881d9d1a7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:17.619948 32066 ts_tablet_manager.cc:1888] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:19:17.620123 32066 log.cc:1196] T 9e13aebe3d3649dda2b2a4f157c08dbc P 6c36b0a7f8a3485f9a9d62e3e7793f25: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/9e13aebe3d3649dda2b2a4f157c08dbc
I20240503 02:19:17.620925 29451 catalog_manager.cc:4830] TS 6c36b0a7f8a3485f9a9d62e3e7793f25 (127.0.145.3:41133): tablet 9e13aebe3d3649dda2b2a4f157c08dbc (table rebalance_test_table_0 [id=dab45517a1ea4baa80835e9059e98803]) successfully deleted
I20240503 02:19:17.628371 30771 tablet_service.cc:1514] Processing DeleteTablet for tablet f83faf506c3e49d4ade5f48881d9d1a7 with delete_type TABLET_DATA_TOMBSTONED (TS 319ccec294484fe9ac61b2e718311b4c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:34112
I20240503 02:19:17.629216 32069 tablet_replica.cc:330] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c: stopping tablet replica
I20240503 02:19:17.629608 32069 raft_consensus.cc:2229] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:17.630033 32069 raft_consensus.cc:2258] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:17.631258 32069 ts_tablet_manager.cc:1875] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:17.635949 32069 ts_tablet_manager.cc:1888] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:19:17.636091 32069 log.cc:1196] T f83faf506c3e49d4ade5f48881d9d1a7 P 319ccec294484fe9ac61b2e718311b4c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/wals/f83faf506c3e49d4ade5f48881d9d1a7
I20240503 02:19:17.636839 29451 catalog_manager.cc:4830] TS 319ccec294484fe9ac61b2e718311b4c (127.0.145.4:39727): tablet f83faf506c3e49d4ade5f48881d9d1a7 (table rebalance_test_table_2 [id=a25280dd90f64d31a487b2c771383e8a]) successfully deleted
I20240503 02:19:17.845921 30771 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:17.855652 29586 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:17.874712 30901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:17.901216 29849 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:17.913650 30640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:17.916743 29716 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:17.921550 31977 raft_consensus.cc:1050] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc: attempting to promote NON_VOTER 319ccec294484fe9ac61b2e718311b4c to VOTER
I20240503 02:19:17.922276 31977 consensus_queue.cc:237] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } }
I20240503 02:19:17.925529 30790 raft_consensus.cc:1261] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c [term 1 LEARNER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:17.925665 29869 raft_consensus.cc:1261] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:17.926214 31977 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:17.926589 30659 raft_consensus.cc:1261] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:17.926788 31977 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:17.928930 31617 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:17.941504 31617 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:17.947676 30790 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:17.947731 29869 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:17.951948 29464 catalog_manager.cc:5484] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: config changed from index 7 to 8, 319ccec294484fe9ac61b2e718311b4c (127.0.145.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:17.958317 29606 consensus_queue.cc:237] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } }
I20240503 02:19:17.962730 29869 raft_consensus.cc:1261] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:19:17.964118 30790 raft_consensus.cc:1261] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Refusing update from remote peer 07bd09ad30cc4932bfff380344c8e9fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:19:17.964936 31617 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:19:17.970527 31617 consensus_queue.cc:1035] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:19:17.971640 31977 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:17.975343 29450 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 548b1357845a405aad60446d2c5d120c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:17.975495 30790 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 319ccec294484fe9ac61b2e718311b4c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:17.976897 29464 catalog_manager.cc:5484] T 548b1357845a405aad60446d2c5d120c P 07bd09ad30cc4932bfff380344c8e9fc reported cstate change: config changed from index 8 to 9, VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) evicted. New cstate: current_term: 1 leader_uuid: "07bd09ad30cc4932bfff380344c8e9fc" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:17.979127 29869 raft_consensus.cc:2932] T 548b1357845a405aad60446d2c5d120c P 6c36b0a7f8a3485f9a9d62e3e7793f25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } }
I20240503 02:19:17.990281 30640 tablet_service.cc:1514] Processing DeleteTablet for tablet 548b1357845a405aad60446d2c5d120c with delete_type TABLET_DATA_TOMBSTONED (TS 58a2474aadf746cc9d50a38f1ee9f80c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33528
I20240503 02:19:17.991366 32132 tablet_replica.cc:330] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: stopping tablet replica
I20240503 02:19:17.991887 32132 raft_consensus.cc:2229] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:17.992455 32132 pending_rounds.cc:62] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: Trying to abort 1 pending ops.
I20240503 02:19:17.992552 32132 pending_rounds.cc:68] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023422496447938560 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "548b1357845a405aad60446d2c5d120c" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: NON_VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07bd09ad30cc4932bfff380344c8e9fc" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 46557 } } peers { permanent_uuid: "6c36b0a7f8a3485f9a9d62e3e7793f25" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 41133 } } peers { permanent_uuid: "58a2474aadf746cc9d50a38f1ee9f80c" member_type: VOTER last_known_addr { host: "127.0.145.6" port: 39629 } attrs { promote: false replace: true } } peers { permanent_uuid: "319ccec294484fe9ac61b2e718311b4c" member_type: VOTER last_known_addr { host: "127.0.145.4" port: 39727 } attrs { promote: false } } } }
I20240503 02:19:17.992952 32132 raft_consensus.cc:2894] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240503 02:19:17.993098 32132 raft_consensus.cc:2866] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:19:17.993208 32132 raft_consensus.cc:2258] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:17.994486 32132 ts_tablet_manager.cc:1875] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:17.999874 32132 ts_tablet_manager.cc:1888] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:19:18.000011 32132 log.cc:1196] T 548b1357845a405aad60446d2c5d120c P 58a2474aadf746cc9d50a38f1ee9f80c: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714702277455189-580-0/raft_consensus-itest-cluster/ts-5/wal/wals/548b1357845a405aad60446d2c5d120c
I20240503 02:19:18.001196 29451 catalog_manager.cc:4830] TS 58a2474aadf746cc9d50a38f1ee9f80c (127.0.145.6:39629): tablet 548b1357845a405aad60446d2c5d120c (table rebalance_test_table_0 [id=dab45517a1ea4baa80835e9059e98803]) successfully deleted
I20240503 02:19:18.093292 29586 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:18.100673 30640 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:18.105113 29716 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:19:18.115743 29849 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:18.116331 30901 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:18.121534 30771 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:18.359014   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 29524
I20240503 02:19:18.367740 29645 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:18.520921   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 29524
W20240503 02:19:18.548352 30593 connection.cc:535] server connection from 127.0.145.1:34843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:19:18.548345 30727 connection.cc:535] server connection from 127.0.145.1:43163 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:19:18.548954   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 29655
W20240503 02:19:18.549098 29802 connection.cc:535] server connection from 127.0.145.1:45577 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:19:18.555243 29777 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:18.691638   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 29655
W20240503 02:19:18.716950 29452 connection.cc:535] server connection from 127.0.145.2:41397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:19:18.716986 29804 connection.cc:535] server connection from 127.0.145.2:49915 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:19:18.717204   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 29786
I20240503 02:19:18.723179 29909 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:18.858745   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 29786
I20240503 02:19:18.884388   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 30709
I20240503 02:19:18.890480 30830 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:19.004252   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 30709
W20240503 02:19:19.026381 29451 connection.cc:535] server connection from 127.0.145.4:46529 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:19:19.026557   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 30839
I20240503 02:19:19.047046 30960 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:19.157872   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 30839
I20240503 02:19:19.178292   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 30578
I20240503 02:19:19.184283 30699 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:19.296768   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 30578
I20240503 02:19:19.319135   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 29434
I20240503 02:19:19.320279 29495 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:19.464412   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 29434
2024-05-03T02:19:19Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3 (23367 ms)
[ RUN      ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2
I20240503 02:19:19.552572   580 test_util.cc:268] Using random seed: 1491133994
I20240503 02:19:19.554445   580 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:19:19.554503   580 ts_itest-base.cc:113] --------------
I20240503 02:19:19.554524   580 ts_itest-base.cc:114] 5 tablet servers
I20240503 02:19:19.554548   580 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:19:19.554615   580 ts_itest-base.cc:116] --------------
2024-05-03T02:19:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:19:19Z Disabled control of system clock
I20240503 02:19:19.602293   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.145.62:35797
--webserver_interface=127.0.145.62
--webserver_port=0
--builtin_ntp_servers=127.0.145.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.145.62:35797
--location_mapping_cmd=/tmp/dist-test-taskfpuZdt/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 02:19:19.938361 32183 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 02:19:19.939100 32183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:19.939294 32183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:19:19.939489 32183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:19.947594 32183 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:19:19.947726 32183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:19:19.947809 32183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:19:19.947883 32183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:19:19.947973 32183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:19:19.957240 32183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskfpuZdt/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.0.145.62:35797
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.145.62:35797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.145.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:19:19.958567 32183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:19:19.960352 32183 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:19:19.973537 32188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:19.974179 32191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:19.974779 32189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:19:19.975320 32183 server_base.cc:1034] running on GCE node
I20240503 02:19:19.976101 32183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:19:19.977455 32183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:19:19.978703 32183 hybrid_clock.cc:648] HybridClock initialized: now 1714702759978623 us; error 55 us; skew 500 ppm
I20240503 02:19:19.979004 32183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:19:19.981091 32183 webserver.cc:460] Webserver started at http://127.0.145.62:35961/ using document root <none> and password file <none>
I20240503 02:19:19.981776 32183 fs_manager.cc:359] Metadata directory not provided
I20240503 02:19:19.981901 32183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:19:19.982218 32183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:19:19.984907 32183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "00e7f80ebf9b48c9be9fdf4eb175d6f6"
format_stamp: "Formatted at 2024-05-03 02:19:19 on dist-test-slave-gq9h"
I20240503 02:19:19.985709 32183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "00e7f80ebf9b48c9be9fdf4eb175d6f6"
format_stamp: "Formatted at 2024-05-03 02:19:19 on dist-test-slave-gq9h"
I20240503 02:19:19.991031 32183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240503 02:19:19.994540 32197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:19.996021 32183 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:19:19.996202 32183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
uuid: "00e7f80ebf9b48c9be9fdf4eb175d6f6"
format_stamp: "Formatted at 2024-05-03 02:19:19 on dist-test-slave-gq9h"
I20240503 02:19:19.996379 32183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:19:20.020007 32183 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:19:20.020884 32183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:19:20.021191 32183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:19:20.039398 32183 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.62:35797
I20240503 02:19:20.039448 32248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.62:35797 every 8 connection(s)
I20240503 02:19:20.040822 32183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:19:20.041817   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 32183
I20240503 02:19:20.042085   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:19:20.045426 32249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:20.058516 32249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6: Bootstrap starting.
I20240503 02:19:20.061316 32249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:20.062314 32249 log.cc:824] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6: Log is configured to *not* fsync() on all Append() calls
I20240503 02:19:20.064736 32249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6: No bootstrap required, opened a new log
I20240503 02:19:20.070735 32249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e7f80ebf9b48c9be9fdf4eb175d6f6" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 35797 } }
I20240503 02:19:20.071090 32249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:20.071192 32249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e7f80ebf9b48c9be9fdf4eb175d6f6, State: Initialized, Role: FOLLOWER
I20240503 02:19:20.071771 32249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e7f80ebf9b48c9be9fdf4eb175d6f6" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 35797 } }
I20240503 02:19:20.071974 32249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:19:20.072098 32249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:19:20.072263 32249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:20.074338 32249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e7f80ebf9b48c9be9fdf4eb175d6f6" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 35797 } }
I20240503 02:19:20.074895 32249 leader_election.cc:302] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00e7f80ebf9b48c9be9fdf4eb175d6f6; no voters: 
I20240503 02:19:20.075493 32249 leader_election.cc:288] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:19:20.075614 32254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:20.076293 32254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [term 1 LEADER]: Becoming Leader. State: Replica: 00e7f80ebf9b48c9be9fdf4eb175d6f6, State: Running, Role: LEADER
I20240503 02:19:20.076860 32254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e7f80ebf9b48c9be9fdf4eb175d6f6" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 35797 } }
I20240503 02:19:20.077623 32249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:19:20.080030 32255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00e7f80ebf9b48c9be9fdf4eb175d6f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e7f80ebf9b48c9be9fdf4eb175d6f6" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 35797 } } }
I20240503 02:19:20.079993 32256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00e7f80ebf9b48c9be9fdf4eb175d6f6. Latest consensus state: current_term: 1 leader_uuid: "00e7f80ebf9b48c9be9fdf4eb175d6f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e7f80ebf9b48c9be9fdf4eb175d6f6" member_type: VOTER last_known_addr { host: "127.0.145.62" port: 35797 } } }
I20240503 02:19:20.080412 32255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [sys.catalog]: This master's current role is: LEADER
I20240503 02:19:20.080418 32256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6 [sys.catalog]: This master's current role is: LEADER
I20240503 02:19:20.087152 32261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:19:20.092628 32261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:19:20.101653 32261 catalog_manager.cc:1347] Generated new cluster ID: 1d407ff7b6cc4faba8e316abb0980d02
I20240503 02:19:20.101789 32261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:19:20.111644 32261 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:19:20.112974 32261 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:19:20.123279 32261 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 00e7f80ebf9b48c9be9fdf4eb175d6f6: Generated new TSK 0
I20240503 02:19:20.124207 32261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:19:20.130039   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.1:0
--local_ip_for_outbound_sockets=127.0.145.1
--webserver_interface=127.0.145.1
--webserver_port=0
--tserver_master_addrs=127.0.145.62:35797
--builtin_ntp_servers=127.0.145.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:19:20.473093 32273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:20.473426 32273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:19:20.473526 32273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:20.480764 32273 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:19:20.480886 32273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:19:20.481021 32273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.1
I20240503 02:19:20.490347 32273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.145.1
--webserver_port=0
--tserver_master_addrs=127.0.145.62:35797
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.1
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:19:20.492007 32273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:19:20.493908 32273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:19:20.507544 32279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:20.507958 32281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:20.508370 32278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:19:20.509210 32273 server_base.cc:1034] running on GCE node
I20240503 02:19:20.509825 32273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:19:20.510830 32273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:19:20.512069 32273 hybrid_clock.cc:648] HybridClock initialized: now 1714702760512017 us; error 45 us; skew 500 ppm
I20240503 02:19:20.512470 32273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:19:20.514938 32273 webserver.cc:460] Webserver started at http://127.0.145.1:43035/ using document root <none> and password file <none>
I20240503 02:19:20.515745 32273 fs_manager.cc:359] Metadata directory not provided
I20240503 02:19:20.515889 32273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:19:20.516234 32273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:19:20.519143 32273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d9296c071f334a2c9078772294a56f25"
format_stamp: "Formatted at 2024-05-03 02:19:20 on dist-test-slave-gq9h"
I20240503 02:19:20.519990 32273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d9296c071f334a2c9078772294a56f25"
format_stamp: "Formatted at 2024-05-03 02:19:20 on dist-test-slave-gq9h"
I20240503 02:19:20.525570 32273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:19:20.528863 32287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:20.530237 32273 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240503 02:19:20.530413 32273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d9296c071f334a2c9078772294a56f25"
format_stamp: "Formatted at 2024-05-03 02:19:20 on dist-test-slave-gq9h"
I20240503 02:19:20.530661 32273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:19:20.564770 32273 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:19:20.565680 32273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:19:20.565982 32273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:19:20.567003 32273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:19:20.568496 32273 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:19:20.568605 32273 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:20.568735 32273 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:19:20.568807 32273 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:20.600093 32273 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.1:45943
I20240503 02:19:20.600147 32399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.1:45943 every 8 connection(s)
I20240503 02:19:20.601508 32273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:19:20.609980   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 32273
I20240503 02:19:20.610234   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:19:20.613559   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.2:0
--local_ip_for_outbound_sockets=127.0.145.2
--webserver_interface=127.0.145.2
--webserver_port=0
--tserver_master_addrs=127.0.145.62:35797
--builtin_ntp_servers=127.0.145.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:19:20.614461 32400 heartbeater.cc:347] Connected to a master server at 127.0.145.62:35797
I20240503 02:19:20.614890 32400 heartbeater.cc:464] Registering TS with master...
I20240503 02:19:20.615954 32400 heartbeater.cc:511] Master 127.0.145.62:35797 requested a full tablet report, sending...
I20240503 02:19:20.679996 32214 ts_manager.cc:194] Registered new tserver with Master: d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943)
I20240503 02:19:20.681679 32214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.1:37587
W20240503 02:19:20.978401 32404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:20.978757 32404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:19:20.978919 32404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:20.986549 32404 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:19:20.986722 32404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:19:20.986899 32404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.2
I20240503 02:19:20.996201 32404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.145.2
--webserver_port=0
--tserver_master_addrs=127.0.145.62:35797
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.2
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:19:20.997525 32404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:19:20.999305 32404 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:19:21.011864 32410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:21.012527 32411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:21.013157 32413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:19:21.013713 32404 server_base.cc:1034] running on GCE node
I20240503 02:19:21.014262 32404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:19:21.015182 32404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:19:21.016377 32404 hybrid_clock.cc:648] HybridClock initialized: now 1714702761016334 us; error 33 us; skew 500 ppm
I20240503 02:19:21.016702 32404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:19:21.019055 32404 webserver.cc:460] Webserver started at http://127.0.145.2:39941/ using document root <none> and password file <none>
I20240503 02:19:21.019805 32404 fs_manager.cc:359] Metadata directory not provided
I20240503 02:19:21.019934 32404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:19:21.020251 32404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:19:21.022873 32404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "37afeff9fa264bc69d4f70185f9665af"
format_stamp: "Formatted at 2024-05-03 02:19:21 on dist-test-slave-gq9h"
I20240503 02:19:21.023631 32404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "37afeff9fa264bc69d4f70185f9665af"
format_stamp: "Formatted at 2024-05-03 02:19:21 on dist-test-slave-gq9h"
I20240503 02:19:21.028817 32404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.002s
I20240503 02:19:21.032094 32419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:21.033334 32404 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240503 02:19:21.033463 32404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "37afeff9fa264bc69d4f70185f9665af"
format_stamp: "Formatted at 2024-05-03 02:19:21 on dist-test-slave-gq9h"
I20240503 02:19:21.033628 32404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:19:21.056537 32404 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:19:21.057435 32404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:19:21.057703 32404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:19:21.058728 32404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:19:21.060292 32404 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:19:21.060392 32404 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:21.060518 32404 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:19:21.060593 32404 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:21.090504 32404 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.2:41993
I20240503 02:19:21.090552 32531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.2:41993 every 8 connection(s)
I20240503 02:19:21.091897 32404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:19:21.101590   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 32404
I20240503 02:19:21.101828   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:19:21.105410   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.3:0
--local_ip_for_outbound_sockets=127.0.145.3
--webserver_interface=127.0.145.3
--webserver_port=0
--tserver_master_addrs=127.0.145.62:35797
--builtin_ntp_servers=127.0.145.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:19:21.105420 32532 heartbeater.cc:347] Connected to a master server at 127.0.145.62:35797
I20240503 02:19:21.105723 32532 heartbeater.cc:464] Registering TS with master...
I20240503 02:19:21.106593 32532 heartbeater.cc:511] Master 127.0.145.62:35797 requested a full tablet report, sending...
I20240503 02:19:21.151556 32214 ts_manager.cc:194] Registered new tserver with Master: 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993)
I20240503 02:19:21.152839 32214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.2:34427
W20240503 02:19:21.444140 32536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:21.444412 32536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:19:21.444519 32536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:21.451567 32536 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:19:21.451704 32536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:19:21.451826 32536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.3
I20240503 02:19:21.460670 32536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.145.3
--webserver_port=0
--tserver_master_addrs=127.0.145.62:35797
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.3
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:19:21.461966 32536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:19:21.463807 32536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:19:21.475967 32542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:21.476693 32541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:21.477150 32544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:19:21.477805 32536 server_base.cc:1034] running on GCE node
I20240503 02:19:21.478466 32536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:19:21.479440 32536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:19:21.480636 32536 hybrid_clock.cc:648] HybridClock initialized: now 1714702761480593 us; error 42 us; skew 500 ppm
I20240503 02:19:21.480929 32536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:19:21.483074 32536 webserver.cc:460] Webserver started at http://127.0.145.3:34297/ using document root <none> and password file <none>
I20240503 02:19:21.483789 32536 fs_manager.cc:359] Metadata directory not provided
I20240503 02:19:21.483901 32536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:19:21.484200 32536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:19:21.486738 32536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7f6faa7decfa4da2bceb8a4aff176d07"
format_stamp: "Formatted at 2024-05-03 02:19:21 on dist-test-slave-gq9h"
I20240503 02:19:21.487478 32536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7f6faa7decfa4da2bceb8a4aff176d07"
format_stamp: "Formatted at 2024-05-03 02:19:21 on dist-test-slave-gq9h"
I20240503 02:19:21.492270 32536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240503 02:19:21.495424 32550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:21.496659 32536 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:19:21.496812 32536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7f6faa7decfa4da2bceb8a4aff176d07"
format_stamp: "Formatted at 2024-05-03 02:19:21 on dist-test-slave-gq9h"
I20240503 02:19:21.497004 32536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:19:21.521039 32536 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:19:21.521890 32536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:19:21.522130 32536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:19:21.523133 32536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:19:21.524514 32536 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:19:21.524613 32536 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:21.524693 32536 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:19:21.524745 32536 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:21.556138 32536 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.3:38283
I20240503 02:19:21.556201 32662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.3:38283 every 8 connection(s)
I20240503 02:19:21.557423 32536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:19:21.559464   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 32536
I20240503 02:19:21.559620   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:19:21.562927   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.4:0
--local_ip_for_outbound_sockets=127.0.145.4
--webserver_interface=127.0.145.4
--webserver_port=0
--tserver_master_addrs=127.0.145.62:35797
--builtin_ntp_servers=127.0.145.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:19:21.570482 32663 heartbeater.cc:347] Connected to a master server at 127.0.145.62:35797
I20240503 02:19:21.570844 32663 heartbeater.cc:464] Registering TS with master...
I20240503 02:19:21.571662 32663 heartbeater.cc:511] Master 127.0.145.62:35797 requested a full tablet report, sending...
I20240503 02:19:21.632709 32214 ts_manager.cc:194] Registered new tserver with Master: 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:21.633644 32214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.3:54165
I20240503 02:19:21.684744 32400 heartbeater.cc:503] Master 127.0.145.62:35797 was elected leader, sending a full tablet report...
W20240503 02:19:21.916697 32666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:21.916996 32666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:19:21.917102 32666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:21.924685 32666 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:19:21.924800 32666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:19:21.924912 32666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.4
I20240503 02:19:21.933457 32666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.145.4
--webserver_port=0
--tserver_master_addrs=127.0.145.62:35797
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.4
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:19:21.934759 32666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:19:21.936488 32666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:19:21.948252 32673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:21.948853 32672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:21.949493 32675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:19:21.950201 32666 server_base.cc:1034] running on GCE node
I20240503 02:19:21.950807 32666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:19:21.951659 32666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:19:21.952849 32666 hybrid_clock.cc:648] HybridClock initialized: now 1714702761952801 us; error 42 us; skew 500 ppm
I20240503 02:19:21.953152 32666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:19:21.955374 32666 webserver.cc:460] Webserver started at http://127.0.145.4:35309/ using document root <none> and password file <none>
I20240503 02:19:21.956072 32666 fs_manager.cc:359] Metadata directory not provided
I20240503 02:19:21.956212 32666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:19:21.956496 32666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:19:21.958930 32666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b5704d091d784c849a96fe720e53cc45"
format_stamp: "Formatted at 2024-05-03 02:19:21 on dist-test-slave-gq9h"
I20240503 02:19:21.959609 32666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b5704d091d784c849a96fe720e53cc45"
format_stamp: "Formatted at 2024-05-03 02:19:21 on dist-test-slave-gq9h"
I20240503 02:19:21.964892 32666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:19:21.969471 32681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:21.971061 32666 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:19:21.971228 32666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b5704d091d784c849a96fe720e53cc45"
format_stamp: "Formatted at 2024-05-03 02:19:21 on dist-test-slave-gq9h"
I20240503 02:19:21.971462 32666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:19:21.988090 32666 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:19:21.988926 32666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:19:21.989205 32666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:19:21.990131 32666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:19:21.991585 32666 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:19:21.991683 32666 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:21.991798 32666 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:19:21.991873 32666 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:22.022526 32666 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.4:37397
I20240503 02:19:22.022572   326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.4:37397 every 8 connection(s)
I20240503 02:19:22.023844 32666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:19:22.027734   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 32666
I20240503 02:19:22.027925   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:19:22.031262   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.5:0
--local_ip_for_outbound_sockets=127.0.145.5
--webserver_interface=127.0.145.5
--webserver_port=0
--tserver_master_addrs=127.0.145.62:35797
--builtin_ntp_servers=127.0.145.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:19:22.037689   327 heartbeater.cc:347] Connected to a master server at 127.0.145.62:35797
I20240503 02:19:22.038048   327 heartbeater.cc:464] Registering TS with master...
I20240503 02:19:22.039108   327 heartbeater.cc:511] Master 127.0.145.62:35797 requested a full tablet report, sending...
I20240503 02:19:22.087267 32214 ts_manager.cc:194] Registered new tserver with Master: b5704d091d784c849a96fe720e53cc45 (127.0.145.4:37397)
I20240503 02:19:22.088222 32214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.4:34143
I20240503 02:19:22.155676 32532 heartbeater.cc:503] Master 127.0.145.62:35797 was elected leader, sending a full tablet report...
W20240503 02:19:22.373890   330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:22.374215   330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:19:22.374320   330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:22.381981   330 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:19:22.382119   330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:19:22.382236   330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.5
I20240503 02:19:22.391338   330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.145.5
--webserver_port=0
--tserver_master_addrs=127.0.145.62:35797
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.5
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:19:22.392738   330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:19:22.394670   330 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:19:22.407610   336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:22.408283   337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:22.408816   339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:19:22.409492   330 server_base.cc:1034] running on GCE node
I20240503 02:19:22.410123   330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:19:22.411159   330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:19:22.412362   330 hybrid_clock.cc:648] HybridClock initialized: now 1714702762412309 us; error 48 us; skew 500 ppm
I20240503 02:19:22.412685   330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:19:22.414999   330 webserver.cc:460] Webserver started at http://127.0.145.5:39607/ using document root <none> and password file <none>
I20240503 02:19:22.415737   330 fs_manager.cc:359] Metadata directory not provided
I20240503 02:19:22.415853   330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:19:22.416167   330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:19:22.418926   330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "8c3494a1fbad4d9aa0aa551169b78fa9"
format_stamp: "Formatted at 2024-05-03 02:19:22 on dist-test-slave-gq9h"
I20240503 02:19:22.419701   330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "8c3494a1fbad4d9aa0aa551169b78fa9"
format_stamp: "Formatted at 2024-05-03 02:19:22 on dist-test-slave-gq9h"
I20240503 02:19:22.424971   330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.001s
I20240503 02:19:22.428586   345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:22.429881   330 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:19:22.430038   330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8c3494a1fbad4d9aa0aa551169b78fa9"
format_stamp: "Formatted at 2024-05-03 02:19:22 on dist-test-slave-gq9h"
I20240503 02:19:22.430244   330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:19:22.453452   330 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:19:22.454361   330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:19:22.454710   330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:19:22.455758   330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:19:22.457330   330 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:19:22.457435   330 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:22.457554   330 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:19:22.457638   330 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:22.489673   330 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.5:41015
I20240503 02:19:22.489725   457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.5:41015 every 8 connection(s)
I20240503 02:19:22.491070   330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:19:22.499847   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 330
I20240503 02:19:22.500034   580 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:19:22.504752   458 heartbeater.cc:347] Connected to a master server at 127.0.145.62:35797
I20240503 02:19:22.505045   458 heartbeater.cc:464] Registering TS with master...
I20240503 02:19:22.505820   458 heartbeater.cc:511] Master 127.0.145.62:35797 requested a full tablet report, sending...
I20240503 02:19:22.549799 32214 ts_manager.cc:194] Registered new tserver with Master: 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5:41015)
I20240503 02:19:22.550767 32214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.5:60907
I20240503 02:19:22.554219   580 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:19:22.572324   580 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:19:22.572494   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 330
I20240503 02:19:22.581219   453 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:22.636528 32663 heartbeater.cc:503] Master 127.0.145.62:35797 was elected leader, sending a full tablet report...
I20240503 02:19:22.664093   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 330
I20240503 02:19:22.680863   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 32666
I20240503 02:19:22.689262   322 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:22.768674   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 32666
I20240503 02:19:26.537364 32214 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35116:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 02:19:26.543805 32214 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:19:26.577739 32596 tablet_service.cc:1467] Processing CreateTablet for tablet e812f1a05cb249f3b087f5f5f6f4c01d (DEFAULT_TABLE table=rebalance_test_table_0 [id=3117cfc63bf2489389e6759839a19b7b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:26.577785 32597 tablet_service.cc:1467] Processing CreateTablet for tablet 10541b8e86304eebbe61b7ebc176eb9d (DEFAULT_TABLE table=rebalance_test_table_0 [id=3117cfc63bf2489389e6759839a19b7b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:26.578337 32598 tablet_service.cc:1467] Processing CreateTablet for tablet d90f1fe288904e879c20a7e9c9d7a140 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3117cfc63bf2489389e6759839a19b7b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:26.579576 32597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10541b8e86304eebbe61b7ebc176eb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:26.580013 32598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90f1fe288904e879c20a7e9c9d7a140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:26.582082 32596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e812f1a05cb249f3b087f5f5f6f4c01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:26.584609 32335 tablet_service.cc:1467] Processing CreateTablet for tablet d90f1fe288904e879c20a7e9c9d7a140 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3117cfc63bf2489389e6759839a19b7b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:26.586189 32335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90f1fe288904e879c20a7e9c9d7a140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:26.589794 32465 tablet_service.cc:1467] Processing CreateTablet for tablet e812f1a05cb249f3b087f5f5f6f4c01d (DEFAULT_TABLE table=rebalance_test_table_0 [id=3117cfc63bf2489389e6759839a19b7b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:26.589897 32466 tablet_service.cc:1467] Processing CreateTablet for tablet 10541b8e86304eebbe61b7ebc176eb9d (DEFAULT_TABLE table=rebalance_test_table_0 [id=3117cfc63bf2489389e6759839a19b7b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:26.590111 32467 tablet_service.cc:1467] Processing CreateTablet for tablet d90f1fe288904e879c20a7e9c9d7a140 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3117cfc63bf2489389e6759839a19b7b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:26.591419 32465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e812f1a05cb249f3b087f5f5f6f4c01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:26.591660 32334 tablet_service.cc:1467] Processing CreateTablet for tablet 10541b8e86304eebbe61b7ebc176eb9d (DEFAULT_TABLE table=rebalance_test_table_0 [id=3117cfc63bf2489389e6759839a19b7b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:26.591868 32467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90f1fe288904e879c20a7e9c9d7a140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:26.592154 32334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10541b8e86304eebbe61b7ebc176eb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:26.593040 32466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10541b8e86304eebbe61b7ebc176eb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:26.594095 32333 tablet_service.cc:1467] Processing CreateTablet for tablet e812f1a05cb249f3b087f5f5f6f4c01d (DEFAULT_TABLE table=rebalance_test_table_0 [id=3117cfc63bf2489389e6759839a19b7b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:26.594537 32333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e812f1a05cb249f3b087f5f5f6f4c01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:26.615553   484 tablet_bootstrap.cc:492] T e812f1a05cb249f3b087f5f5f6f4c01d P 7f6faa7decfa4da2bceb8a4aff176d07: Bootstrap starting.
I20240503 02:19:26.615862   483 tablet_bootstrap.cc:492] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25: Bootstrap starting.
I20240503 02:19:26.619200   482 tablet_bootstrap.cc:492] T e812f1a05cb249f3b087f5f5f6f4c01d P 37afeff9fa264bc69d4f70185f9665af: Bootstrap starting.
I20240503 02:19:26.619750   483 tablet_bootstrap.cc:654] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:26.620345   484 tablet_bootstrap.cc:654] T e812f1a05cb249f3b087f5f5f6f4c01d P 7f6faa7decfa4da2bceb8a4aff176d07: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:26.621093   483 log.cc:824] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25: Log is configured to *not* fsync() on all Append() calls
I20240503 02:19:26.621315   484 log.cc:824] T e812f1a05cb249f3b087f5f5f6f4c01d P 7f6faa7decfa4da2bceb8a4aff176d07: Log is configured to *not* fsync() on all Append() calls
I20240503 02:19:26.623271   482 tablet_bootstrap.cc:654] T e812f1a05cb249f3b087f5f5f6f4c01d P 37afeff9fa264bc69d4f70185f9665af: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:26.624356   484 tablet_bootstrap.cc:492] T e812f1a05cb249f3b087f5f5f6f4c01d P 7f6faa7decfa4da2bceb8a4aff176d07: No bootstrap required, opened a new log
I20240503 02:19:26.624485   482 log.cc:824] T e812f1a05cb249f3b087f5f5f6f4c01d P 37afeff9fa264bc69d4f70185f9665af: Log is configured to *not* fsync() on all Append() calls
I20240503 02:19:26.624610   483 tablet_bootstrap.cc:492] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25: No bootstrap required, opened a new log
I20240503 02:19:26.624614   484 ts_tablet_manager.cc:1367] T e812f1a05cb249f3b087f5f5f6f4c01d P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240503 02:19:26.624866   483 ts_tablet_manager.cc:1367] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240503 02:19:26.629168   482 tablet_bootstrap.cc:492] T e812f1a05cb249f3b087f5f5f6f4c01d P 37afeff9fa264bc69d4f70185f9665af: No bootstrap required, opened a new log
I20240503 02:19:26.629462   482 ts_tablet_manager.cc:1367] T e812f1a05cb249f3b087f5f5f6f4c01d P 37afeff9fa264bc69d4f70185f9665af: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.003s
I20240503 02:19:26.631057   484 raft_consensus.cc:348] T e812f1a05cb249f3b087f5f5f6f4c01d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.631414   484 raft_consensus.cc:374] T e812f1a05cb249f3b087f5f5f6f4c01d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:26.631511   484 raft_consensus.cc:729] T e812f1a05cb249f3b087f5f5f6f4c01d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6faa7decfa4da2bceb8a4aff176d07, State: Initialized, Role: FOLLOWER
I20240503 02:19:26.632143   484 consensus_queue.cc:260] T e812f1a05cb249f3b087f5f5f6f4c01d P 7f6faa7decfa4da2bceb8a4aff176d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.633711   484 ts_tablet_manager.cc:1398] T e812f1a05cb249f3b087f5f5f6f4c01d P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
I20240503 02:19:26.634096   484 tablet_bootstrap.cc:492] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: Bootstrap starting.
I20240503 02:19:26.633903   483 raft_consensus.cc:348] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.634402   483 raft_consensus.cc:374] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:26.634550   483 raft_consensus.cc:729] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9296c071f334a2c9078772294a56f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:26.635768   483 consensus_queue.cc:260] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.636351   484 tablet_bootstrap.cc:654] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:26.637559   483 ts_tablet_manager.cc:1398] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25: Time spent starting tablet: real 0.013s	user 0.011s	sys 0.000s
I20240503 02:19:26.638023   483 tablet_bootstrap.cc:492] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25: Bootstrap starting.
I20240503 02:19:26.638358   484 tablet_bootstrap.cc:492] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: No bootstrap required, opened a new log
I20240503 02:19:26.638504   484 ts_tablet_manager.cc:1367] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:19:26.639385   484 raft_consensus.cc:348] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.639572   484 raft_consensus.cc:374] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:26.639678   484 raft_consensus.cc:729] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6faa7decfa4da2bceb8a4aff176d07, State: Initialized, Role: FOLLOWER
I20240503 02:19:26.639796   483 tablet_bootstrap.cc:654] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:26.639989   484 consensus_queue.cc:260] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.640456   484 ts_tablet_manager.cc:1398] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:19:26.640712   484 tablet_bootstrap.cc:492] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07: Bootstrap starting.
I20240503 02:19:26.642832   484 tablet_bootstrap.cc:654] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:26.648041   483 tablet_bootstrap.cc:492] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25: No bootstrap required, opened a new log
I20240503 02:19:26.648164   484 tablet_bootstrap.cc:492] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07: No bootstrap required, opened a new log
I20240503 02:19:26.648238   483 ts_tablet_manager.cc:1367] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25: Time spent bootstrapping tablet: real 0.010s	user 0.009s	sys 0.000s
I20240503 02:19:26.648313   484 ts_tablet_manager.cc:1367] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.004s
I20240503 02:19:26.649009   484 raft_consensus.cc:348] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } }
I20240503 02:19:26.649253   484 raft_consensus.cc:374] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:26.649346   484 raft_consensus.cc:729] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6faa7decfa4da2bceb8a4aff176d07, State: Initialized, Role: FOLLOWER
I20240503 02:19:26.649248   483 raft_consensus.cc:348] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.649463   483 raft_consensus.cc:374] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:26.649304   482 raft_consensus.cc:348] T e812f1a05cb249f3b087f5f5f6f4c01d P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.649667   483 raft_consensus.cc:729] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9296c071f334a2c9078772294a56f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:26.649618   484 consensus_queue.cc:260] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } }
I20240503 02:19:26.649747   482 raft_consensus.cc:374] T e812f1a05cb249f3b087f5f5f6f4c01d P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:26.649857   482 raft_consensus.cc:729] T e812f1a05cb249f3b087f5f5f6f4c01d P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Initialized, Role: FOLLOWER
I20240503 02:19:26.649904   483 consensus_queue.cc:260] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.650334   484 ts_tablet_manager.cc:1398] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:19:26.650447   483 ts_tablet_manager.cc:1398] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:19:26.650738   483 tablet_bootstrap.cc:492] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25: Bootstrap starting.
I20240503 02:19:26.652330   483 tablet_bootstrap.cc:654] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:26.653136   482 consensus_queue.cc:260] T e812f1a05cb249f3b087f5f5f6f4c01d P 37afeff9fa264bc69d4f70185f9665af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.654265   483 tablet_bootstrap.cc:492] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25: No bootstrap required, opened a new log
I20240503 02:19:26.654390   483 ts_tablet_manager.cc:1367] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:19:26.655011   483 raft_consensus.cc:348] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } }
I20240503 02:19:26.655197   483 raft_consensus.cc:374] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:26.655280   483 raft_consensus.cc:729] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9296c071f334a2c9078772294a56f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:26.655520   483 consensus_queue.cc:260] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } }
I20240503 02:19:26.656016   483 ts_tablet_manager.cc:1398] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:19:26.657464   482 ts_tablet_manager.cc:1398] T e812f1a05cb249f3b087f5f5f6f4c01d P 37afeff9fa264bc69d4f70185f9665af: Time spent starting tablet: real 0.028s	user 0.011s	sys 0.005s
I20240503 02:19:26.657876   482 tablet_bootstrap.cc:492] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af: Bootstrap starting.
I20240503 02:19:26.659569   482 tablet_bootstrap.cc:654] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:26.661613   482 tablet_bootstrap.cc:492] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af: No bootstrap required, opened a new log
I20240503 02:19:26.661764   482 ts_tablet_manager.cc:1367] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 02:19:26.662483   482 raft_consensus.cc:348] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } }
I20240503 02:19:26.662664   482 raft_consensus.cc:374] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:26.662767   482 raft_consensus.cc:729] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Initialized, Role: FOLLOWER
I20240503 02:19:26.662969   482 consensus_queue.cc:260] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } }
I20240503 02:19:26.663394   482 ts_tablet_manager.cc:1398] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:19:26.663715   482 tablet_bootstrap.cc:492] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: Bootstrap starting.
I20240503 02:19:26.665022   482 tablet_bootstrap.cc:654] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:26.666913   482 tablet_bootstrap.cc:492] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: No bootstrap required, opened a new log
I20240503 02:19:26.667044   482 ts_tablet_manager.cc:1367] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: Time spent bootstrapping tablet: real 0.003s	user 0.000s	sys 0.003s
I20240503 02:19:26.667529   482 raft_consensus.cc:348] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.667695   482 raft_consensus.cc:374] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:26.667748   482 raft_consensus.cc:729] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Initialized, Role: FOLLOWER
I20240503 02:19:26.667943   482 consensus_queue.cc:260] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.668404   482 ts_tablet_manager.cc:1398] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:19:26.758937   494 raft_consensus.cc:482] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:26.759267   494 raft_consensus.cc:504] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.761116   494 leader_election.cc:288] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:26.764621   494 raft_consensus.cc:482] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:26.764807   494 raft_consensus.cc:504] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } }
I20240503 02:19:26.765470   494 leader_election.cc:288] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:26.769310 32355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10541b8e86304eebbe61b7ebc176eb9d" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9296c071f334a2c9078772294a56f25" is_pre_election: true
I20240503 02:19:26.769289 32354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90f1fe288904e879c20a7e9c9d7a140" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9296c071f334a2c9078772294a56f25" is_pre_election: true
I20240503 02:19:26.769445 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90f1fe288904e879c20a7e9c9d7a140" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" is_pre_election: true
I20240503 02:19:26.769423 32618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10541b8e86304eebbe61b7ebc176eb9d" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" is_pre_election: true
I20240503 02:19:26.770046 32355 raft_consensus.cc:2454] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 0.
I20240503 02:19:26.770097 32617 raft_consensus.cc:2454] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 0.
I20240503 02:19:26.770097 32618 raft_consensus.cc:2454] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 0.
I20240503 02:19:26.770046 32354 raft_consensus.cc:2454] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 0.
I20240503 02:19:26.770768 32421 leader_election.cc:302] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37afeff9fa264bc69d4f70185f9665af, 7f6faa7decfa4da2bceb8a4aff176d07; no voters: 
I20240503 02:19:26.770745 32420 leader_election.cc:302] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37afeff9fa264bc69d4f70185f9665af, d9296c071f334a2c9078772294a56f25; no voters: 
I20240503 02:19:26.771219   494 raft_consensus.cc:2781] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:26.771396   494 raft_consensus.cc:482] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:26.771543   494 raft_consensus.cc:3037] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:26.771695   499 raft_consensus.cc:2781] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:26.771812   499 raft_consensus.cc:482] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:26.771950   499 raft_consensus.cc:3037] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:26.773939   494 raft_consensus.cc:504] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } }
I20240503 02:19:26.774012   499 raft_consensus.cc:504] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.774642   499 leader_election.cc:288] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 election: Requested vote from peers d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:26.774670   494 leader_election.cc:288] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 election: Requested vote from peers d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:26.775229 32618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10541b8e86304eebbe61b7ebc176eb9d" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07"
I20240503 02:19:26.775300 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90f1fe288904e879c20a7e9c9d7a140" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07"
I20240503 02:19:26.775312 32354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90f1fe288904e879c20a7e9c9d7a140" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9296c071f334a2c9078772294a56f25"
I20240503 02:19:26.775434 32618 raft_consensus.cc:3037] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:26.775501 32617 raft_consensus.cc:3037] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:26.775499 32354 raft_consensus.cc:3037] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:26.775954 32355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10541b8e86304eebbe61b7ebc176eb9d" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9296c071f334a2c9078772294a56f25"
I20240503 02:19:26.776132 32355 raft_consensus.cc:3037] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:26.778803 32617 raft_consensus.cc:2454] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 1.
I20240503 02:19:26.778848 32618 raft_consensus.cc:2454] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 1.
I20240503 02:19:26.779248 32355 raft_consensus.cc:2454] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 1.
I20240503 02:19:26.779376 32421 leader_election.cc:302] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37afeff9fa264bc69d4f70185f9665af, 7f6faa7decfa4da2bceb8a4aff176d07; no voters: 
I20240503 02:19:26.779732 32421 leader_election.cc:302] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37afeff9fa264bc69d4f70185f9665af, 7f6faa7decfa4da2bceb8a4aff176d07; no voters: 
I20240503 02:19:26.780009   494 raft_consensus.cc:2781] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:26.780246   499 raft_consensus.cc:2781] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:26.780678   494 raft_consensus.cc:686] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Becoming Leader. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Running, Role: LEADER
I20240503 02:19:26.780987 32354 raft_consensus.cc:2454] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 1.
I20240503 02:19:26.781286   494 consensus_queue.cc:237] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } }
I20240503 02:19:26.781550   499 raft_consensus.cc:686] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Becoming Leader. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Running, Role: LEADER
I20240503 02:19:26.781795   499 consensus_queue.cc:237] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.786754 32213 catalog_manager.cc:5484] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af reported cstate change: term changed from 0 to 1, leader changed from <none> to 37afeff9fa264bc69d4f70185f9665af (127.0.145.2). New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } health_report { overall_health: HEALTHY } } }
I20240503 02:19:26.791682 32213 catalog_manager.cc:5484] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af reported cstate change: term changed from 0 to 1, leader changed from <none> to 37afeff9fa264bc69d4f70185f9665af (127.0.145.2). New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: UNKNOWN } } }
W20240503 02:19:26.813313 32664 tablet.cc:2369] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:19:26.848381 32533 tablet.cc:2369] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:19:26.848908   489 raft_consensus.cc:482] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:26.849102   489 raft_consensus.cc:504] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.850735   489 leader_election.cc:288] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:26.857386 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e812f1a05cb249f3b087f5f5f6f4c01d" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" is_pre_election: true
I20240503 02:19:26.857648 32617 raft_consensus.cc:2454] T e812f1a05cb249f3b087f5f5f6f4c01d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 0.
I20240503 02:19:26.857532 32487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e812f1a05cb249f3b087f5f5f6f4c01d" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37afeff9fa264bc69d4f70185f9665af" is_pre_election: true
W20240503 02:19:26.857753 32401 tablet.cc:2369] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:19:26.857988 32487 raft_consensus.cc:2454] T e812f1a05cb249f3b087f5f5f6f4c01d P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 0.
I20240503 02:19:26.858078 32289 leader_election.cc:302] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6faa7decfa4da2bceb8a4aff176d07, d9296c071f334a2c9078772294a56f25; no voters: 
I20240503 02:19:26.858485   489 raft_consensus.cc:2781] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:26.858640   489 raft_consensus.cc:482] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:26.858747   489 raft_consensus.cc:3037] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:26.860693   489 raft_consensus.cc:504] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.861270   489 leader_election.cc:288] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [CANDIDATE]: Term 1 election: Requested vote from peers 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:26.861639 32487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e812f1a05cb249f3b087f5f5f6f4c01d" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37afeff9fa264bc69d4f70185f9665af"
I20240503 02:19:26.861732 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e812f1a05cb249f3b087f5f5f6f4c01d" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07"
I20240503 02:19:26.861801 32487 raft_consensus.cc:3037] T e812f1a05cb249f3b087f5f5f6f4c01d P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:26.861935 32617 raft_consensus.cc:3037] T e812f1a05cb249f3b087f5f5f6f4c01d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:26.863710 32617 raft_consensus.cc:2454] T e812f1a05cb249f3b087f5f5f6f4c01d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 1.
I20240503 02:19:26.863729 32487 raft_consensus.cc:2454] T e812f1a05cb249f3b087f5f5f6f4c01d P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 1.
I20240503 02:19:26.864091 32289 leader_election.cc:302] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37afeff9fa264bc69d4f70185f9665af, d9296c071f334a2c9078772294a56f25; no voters: 
I20240503 02:19:26.864346   489 raft_consensus.cc:2781] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:26.864751   489 raft_consensus.cc:686] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [term 1 LEADER]: Becoming Leader. State: Replica: d9296c071f334a2c9078772294a56f25, State: Running, Role: LEADER
I20240503 02:19:26.865301   489 consensus_queue.cc:237] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:26.869325 32213 catalog_manager.cc:5484] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9296c071f334a2c9078772294a56f25 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "d9296c071f334a2c9078772294a56f25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:27.188889   499 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:27.197589   499 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:27.265264   499 consensus_queue.cc:1035] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:27.270470   494 consensus_queue.cc:1035] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:19:27.275275   508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:27.275656   508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:27.283504   508 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 02:19:27.318645   489 consensus_queue.cc:1035] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:27.325295   506 consensus_queue.cc:1035] T e812f1a05cb249f3b087f5f5f6f4c01d P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 223.008 ms
  time per row: 0.111504 ms
I20240503 02:19:27.656672 32208 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35116:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 02:19:27.657438 32208 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:19:27.665668 32465 tablet_service.cc:1467] Processing CreateTablet for tablet 991ee5bce8494e908449e49097e14e45 (DEFAULT_TABLE table=rebalance_test_table_1 [id=71ee030fc6ed4819be1bd2168142c3a4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:27.665995 32596 tablet_service.cc:1467] Processing CreateTablet for tablet 991ee5bce8494e908449e49097e14e45 (DEFAULT_TABLE table=rebalance_test_table_1 [id=71ee030fc6ed4819be1bd2168142c3a4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:27.666199 32465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991ee5bce8494e908449e49097e14e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:27.666262 32334 tablet_service.cc:1467] Processing CreateTablet for tablet 991ee5bce8494e908449e49097e14e45 (DEFAULT_TABLE table=rebalance_test_table_1 [id=71ee030fc6ed4819be1bd2168142c3a4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:27.666405 32596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991ee5bce8494e908449e49097e14e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:27.666664 32334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991ee5bce8494e908449e49097e14e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:27.667649 32335 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1f6a5ac64a46b09d4bbd936260cb3f (DEFAULT_TABLE table=rebalance_test_table_1 [id=71ee030fc6ed4819be1bd2168142c3a4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:27.668078 32335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1f6a5ac64a46b09d4bbd936260cb3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:27.671062 32333 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1f43f03ccf472c821943d848064d63 (DEFAULT_TABLE table=rebalance_test_table_1 [id=71ee030fc6ed4819be1bd2168142c3a4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:27.671541 32333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1f43f03ccf472c821943d848064d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:27.673792 32598 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1f6a5ac64a46b09d4bbd936260cb3f (DEFAULT_TABLE table=rebalance_test_table_1 [id=71ee030fc6ed4819be1bd2168142c3a4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:27.674301 32598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1f6a5ac64a46b09d4bbd936260cb3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:27.678133 32597 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1f43f03ccf472c821943d848064d63 (DEFAULT_TABLE table=rebalance_test_table_1 [id=71ee030fc6ed4819be1bd2168142c3a4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:27.679311 32597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1f43f03ccf472c821943d848064d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:27.679809 32467 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1f6a5ac64a46b09d4bbd936260cb3f (DEFAULT_TABLE table=rebalance_test_table_1 [id=71ee030fc6ed4819be1bd2168142c3a4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:27.680251 32467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1f6a5ac64a46b09d4bbd936260cb3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:27.681888   572 tablet_bootstrap.cc:492] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25: Bootstrap starting.
I20240503 02:19:27.683696   572 tablet_bootstrap.cc:654] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:27.685576 32466 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1f43f03ccf472c821943d848064d63 (DEFAULT_TABLE table=rebalance_test_table_1 [id=71ee030fc6ed4819be1bd2168142c3a4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:27.686012 32466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1f43f03ccf472c821943d848064d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:27.678980   570 tablet_bootstrap.cc:492] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07: Bootstrap starting.
I20240503 02:19:27.689699   570 tablet_bootstrap.cc:654] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:27.692018   572 tablet_bootstrap.cc:492] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25: No bootstrap required, opened a new log
I20240503 02:19:27.692195   572 ts_tablet_manager.cc:1367] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.002s
I20240503 02:19:27.692927   572 raft_consensus.cc:348] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.693107   572 raft_consensus.cc:374] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:27.693195   572 raft_consensus.cc:729] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9296c071f334a2c9078772294a56f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:27.693435   572 consensus_queue.cc:260] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.693943   572 ts_tablet_manager.cc:1398] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:19:27.694192   572 tablet_bootstrap.cc:492] T 9f1f6a5ac64a46b09d4bbd936260cb3f P d9296c071f334a2c9078772294a56f25: Bootstrap starting.
I20240503 02:19:27.696049   572 tablet_bootstrap.cc:654] T 9f1f6a5ac64a46b09d4bbd936260cb3f P d9296c071f334a2c9078772294a56f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:27.697121   571 tablet_bootstrap.cc:492] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af: Bootstrap starting.
I20240503 02:19:27.699221   570 tablet_bootstrap.cc:492] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07: No bootstrap required, opened a new log
I20240503 02:19:27.699405   570 ts_tablet_manager.cc:1367] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent bootstrapping tablet: real 0.021s	user 0.004s	sys 0.000s
I20240503 02:19:27.699621   571 tablet_bootstrap.cc:654] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:27.700452   570 raft_consensus.cc:348] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.700803   570 raft_consensus.cc:374] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:27.700965   570 raft_consensus.cc:729] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6faa7decfa4da2bceb8a4aff176d07, State: Initialized, Role: FOLLOWER
I20240503 02:19:27.701265   570 consensus_queue.cc:260] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.702349   571 tablet_bootstrap.cc:492] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af: No bootstrap required, opened a new log
I20240503 02:19:27.702527   571 ts_tablet_manager.cc:1367] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240503 02:19:27.702633   570 ts_tablet_manager.cc:1398] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:19:27.702977   570 tablet_bootstrap.cc:492] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 7f6faa7decfa4da2bceb8a4aff176d07: Bootstrap starting.
I20240503 02:19:27.703436   571 raft_consensus.cc:348] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.703698   571 raft_consensus.cc:374] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:27.703814   571 raft_consensus.cc:729] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Initialized, Role: FOLLOWER
I20240503 02:19:27.704113   571 consensus_queue.cc:260] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.704735   571 ts_tablet_manager.cc:1398] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:19:27.704988   570 tablet_bootstrap.cc:654] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 7f6faa7decfa4da2bceb8a4aff176d07: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:27.705099   571 tablet_bootstrap.cc:492] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: Bootstrap starting.
I20240503 02:19:27.707230   570 tablet_bootstrap.cc:492] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 7f6faa7decfa4da2bceb8a4aff176d07: No bootstrap required, opened a new log
I20240503 02:19:27.707247   572 tablet_bootstrap.cc:492] T 9f1f6a5ac64a46b09d4bbd936260cb3f P d9296c071f334a2c9078772294a56f25: No bootstrap required, opened a new log
I20240503 02:19:27.707417   570 ts_tablet_manager.cc:1367] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 02:19:27.707566   572 ts_tablet_manager.cc:1367] T 9f1f6a5ac64a46b09d4bbd936260cb3f P d9296c071f334a2c9078772294a56f25: Time spent bootstrapping tablet: real 0.013s	user 0.000s	sys 0.006s
I20240503 02:19:27.707906   571 tablet_bootstrap.cc:654] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:27.708492   572 raft_consensus.cc:348] T 9f1f6a5ac64a46b09d4bbd936260cb3f P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.708781   572 raft_consensus.cc:374] T 9f1f6a5ac64a46b09d4bbd936260cb3f P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:27.708906   572 raft_consensus.cc:729] T 9f1f6a5ac64a46b09d4bbd936260cb3f P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9296c071f334a2c9078772294a56f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:27.709216   572 consensus_queue.cc:260] T 9f1f6a5ac64a46b09d4bbd936260cb3f P d9296c071f334a2c9078772294a56f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.709520   570 raft_consensus.cc:348] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.709761   570 raft_consensus.cc:374] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:27.709877   570 raft_consensus.cc:729] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6faa7decfa4da2bceb8a4aff176d07, State: Initialized, Role: FOLLOWER
I20240503 02:19:27.710137   570 consensus_queue.cc:260] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 7f6faa7decfa4da2bceb8a4aff176d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.710359   571 tablet_bootstrap.cc:492] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: No bootstrap required, opened a new log
I20240503 02:19:27.710510   572 ts_tablet_manager.cc:1398] T 9f1f6a5ac64a46b09d4bbd936260cb3f P d9296c071f334a2c9078772294a56f25: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 02:19:27.710527   571 ts_tablet_manager.cc:1367] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240503 02:19:27.710808   572 tablet_bootstrap.cc:492] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25: Bootstrap starting.
I20240503 02:19:27.711455   571 raft_consensus.cc:348] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.711696   571 raft_consensus.cc:374] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:27.711779   571 raft_consensus.cc:729] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Initialized, Role: FOLLOWER
I20240503 02:19:27.712015   570 ts_tablet_manager.cc:1398] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.001s
I20240503 02:19:27.712090   572 tablet_bootstrap.cc:654] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:27.712055   571 consensus_queue.cc:260] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.712350   570 tablet_bootstrap.cc:492] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07: Bootstrap starting.
I20240503 02:19:27.712735   571 ts_tablet_manager.cc:1398] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:19:27.713088   571 tablet_bootstrap.cc:492] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af: Bootstrap starting.
I20240503 02:19:27.714867   571 tablet_bootstrap.cc:654] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:27.716405   570 tablet_bootstrap.cc:654] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:27.723448   570 tablet_bootstrap.cc:492] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07: No bootstrap required, opened a new log
I20240503 02:19:27.723644   570 ts_tablet_manager.cc:1367] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.000s
I20240503 02:19:27.723731   571 tablet_bootstrap.cc:492] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af: No bootstrap required, opened a new log
I20240503 02:19:27.723915   571 ts_tablet_manager.cc:1367] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af: Time spent bootstrapping tablet: real 0.011s	user 0.000s	sys 0.009s
I20240503 02:19:27.723932   572 tablet_bootstrap.cc:492] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25: No bootstrap required, opened a new log
I20240503 02:19:27.724083   572 ts_tablet_manager.cc:1367] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25: Time spent bootstrapping tablet: real 0.013s	user 0.004s	sys 0.007s
I20240503 02:19:27.724363   499 raft_consensus.cc:482] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:27.724536   499 raft_consensus.cc:504] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.724574   570 raft_consensus.cc:348] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.724792   570 raft_consensus.cc:374] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:27.724889   570 raft_consensus.cc:729] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6faa7decfa4da2bceb8a4aff176d07, State: Initialized, Role: FOLLOWER
I20240503 02:19:27.724802   571 raft_consensus.cc:348] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.724905   572 raft_consensus.cc:348] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.725020   571 raft_consensus.cc:374] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:27.725078   572 raft_consensus.cc:374] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:27.725116   571 raft_consensus.cc:729] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Initialized, Role: FOLLOWER
I20240503 02:19:27.725179   572 raft_consensus.cc:729] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9296c071f334a2c9078772294a56f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:27.725140   570 consensus_queue.cc:260] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.725286   499 leader_election.cc:288] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:27.725371   571 consensus_queue.cc:260] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.725883 32355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991ee5bce8494e908449e49097e14e45" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9296c071f334a2c9078772294a56f25" is_pre_election: true
I20240503 02:19:27.725919   570 ts_tablet_manager.cc:1398] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 02:19:27.725912 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991ee5bce8494e908449e49097e14e45" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" is_pre_election: true
I20240503 02:19:27.726097 32355 raft_consensus.cc:2454] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 0.
I20240503 02:19:27.726164 32617 raft_consensus.cc:2454] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 0.
I20240503 02:19:27.726442   572 consensus_queue.cc:260] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.726754 32421 leader_election.cc:302] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37afeff9fa264bc69d4f70185f9665af, 7f6faa7decfa4da2bceb8a4aff176d07; no voters: 
I20240503 02:19:27.726989   572 ts_tablet_manager.cc:1398] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 02:19:27.727185   499 raft_consensus.cc:2781] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:27.727277   499 raft_consensus.cc:482] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:27.727381   499 raft_consensus.cc:3037] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:27.727612   571 ts_tablet_manager.cc:1398] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.002s
I20240503 02:19:27.731446   499 raft_consensus.cc:504] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.732246   499 leader_election.cc:288] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 election: Requested vote from peers d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:27.732743 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991ee5bce8494e908449e49097e14e45" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07"
I20240503 02:19:27.732980 32617 raft_consensus.cc:3037] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:27.733033 32355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991ee5bce8494e908449e49097e14e45" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9296c071f334a2c9078772294a56f25"
I20240503 02:19:27.733255 32355 raft_consensus.cc:3037] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:27.735190 32617 raft_consensus.cc:2454] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 1.
I20240503 02:19:27.735204 32355 raft_consensus.cc:2454] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 1.
I20240503 02:19:27.735626 32421 leader_election.cc:302] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37afeff9fa264bc69d4f70185f9665af, 7f6faa7decfa4da2bceb8a4aff176d07; no voters: 
I20240503 02:19:27.735915   499 raft_consensus.cc:2781] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:27.736092   499 raft_consensus.cc:686] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Becoming Leader. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Running, Role: LEADER
I20240503 02:19:27.736330   499 consensus_queue.cc:237] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.738941 32209 catalog_manager.cc:5484] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af reported cstate change: term changed from 0 to 1, leader changed from <none> to 37afeff9fa264bc69d4f70185f9665af (127.0.145.2). New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:27.793526   499 raft_consensus.cc:482] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:27.793745   499 raft_consensus.cc:504] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.794421   499 leader_election.cc:288] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:27.794927 32355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1f6a5ac64a46b09d4bbd936260cb3f" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9296c071f334a2c9078772294a56f25" is_pre_election: true
I20240503 02:19:27.795042 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1f6a5ac64a46b09d4bbd936260cb3f" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" is_pre_election: true
I20240503 02:19:27.795200 32355 raft_consensus.cc:2454] T 9f1f6a5ac64a46b09d4bbd936260cb3f P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 0.
I20240503 02:19:27.795279 32617 raft_consensus.cc:2454] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 0.
I20240503 02:19:27.795701 32420 leader_election.cc:302] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37afeff9fa264bc69d4f70185f9665af, d9296c071f334a2c9078772294a56f25; no voters: 
I20240503 02:19:27.795944   499 raft_consensus.cc:2781] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:27.796056   499 raft_consensus.cc:482] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:27.796144   499 raft_consensus.cc:3037] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:27.798009   499 raft_consensus.cc:504] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.798518   499 leader_election.cc:288] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 election: Requested vote from peers d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:27.799031 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1f6a5ac64a46b09d4bbd936260cb3f" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07"
I20240503 02:19:27.799062 32355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1f6a5ac64a46b09d4bbd936260cb3f" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9296c071f334a2c9078772294a56f25"
I20240503 02:19:27.799247 32355 raft_consensus.cc:3037] T 9f1f6a5ac64a46b09d4bbd936260cb3f P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:27.799269 32617 raft_consensus.cc:3037] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:27.802142 32617 raft_consensus.cc:2454] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 1.
I20240503 02:19:27.802155 32355 raft_consensus.cc:2454] T 9f1f6a5ac64a46b09d4bbd936260cb3f P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 1.
I20240503 02:19:27.802642 32420 leader_election.cc:302] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37afeff9fa264bc69d4f70185f9665af, d9296c071f334a2c9078772294a56f25; no voters: 
I20240503 02:19:27.802951   499 raft_consensus.cc:2781] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:27.803099   499 raft_consensus.cc:686] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Becoming Leader. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Running, Role: LEADER
I20240503 02:19:27.803361   499 consensus_queue.cc:237] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.806010 32209 catalog_manager.cc:5484] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af reported cstate change: term changed from 0 to 1, leader changed from <none> to 37afeff9fa264bc69d4f70185f9665af (127.0.145.2). New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:27.867262   506 raft_consensus.cc:482] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:27.867467   506 raft_consensus.cc:504] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.868086   506 leader_election.cc:288] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:27.868614 32487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1f43f03ccf472c821943d848064d63" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37afeff9fa264bc69d4f70185f9665af" is_pre_election: true
I20240503 02:19:27.868710 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1f43f03ccf472c821943d848064d63" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" is_pre_election: true
I20240503 02:19:27.868878 32487 raft_consensus.cc:2454] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 0.
I20240503 02:19:27.868901 32617 raft_consensus.cc:2454] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 0.
I20240503 02:19:27.869316 32289 leader_election.cc:302] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6faa7decfa4da2bceb8a4aff176d07, d9296c071f334a2c9078772294a56f25; no voters: 
I20240503 02:19:27.869571   506 raft_consensus.cc:2781] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:27.869659   506 raft_consensus.cc:482] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:27.869737   506 raft_consensus.cc:3037] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:27.871722   506 raft_consensus.cc:504] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.872277   506 leader_election.cc:288] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [CANDIDATE]: Term 1 election: Requested vote from peers 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:27.872767 32487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1f43f03ccf472c821943d848064d63" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37afeff9fa264bc69d4f70185f9665af"
I20240503 02:19:27.872854 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1f43f03ccf472c821943d848064d63" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07"
I20240503 02:19:27.872962 32487 raft_consensus.cc:3037] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:27.873080 32617 raft_consensus.cc:3037] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:27.874930 32487 raft_consensus.cc:2454] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 1.
I20240503 02:19:27.875089 32617 raft_consensus.cc:2454] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 1.
I20240503 02:19:27.875348 32289 leader_election.cc:302] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37afeff9fa264bc69d4f70185f9665af, d9296c071f334a2c9078772294a56f25; no voters: 
I20240503 02:19:27.875648   506 raft_consensus.cc:2781] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:27.875805   506 raft_consensus.cc:686] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 1 LEADER]: Becoming Leader. State: Replica: d9296c071f334a2c9078772294a56f25, State: Running, Role: LEADER
I20240503 02:19:27.876106   506 consensus_queue.cc:237] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:27.878989 32209 catalog_manager.cc:5484] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9296c071f334a2c9078772294a56f25 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "d9296c071f334a2c9078772294a56f25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:28.144320   499 consensus_queue.cc:1035] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:28.149838   499 consensus_queue.cc:1035] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:28.217190   499 consensus_queue.cc:1035] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:28.221863   501 consensus_queue.cc:1035] T 9f1f6a5ac64a46b09d4bbd936260cb3f P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:19:28.289088   590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:28.289461   590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:28.296919   590 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:19:28.341933   506 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:28.347648   506 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 197.541 ms
  time per row: 0.0987706 ms
I20240503 02:19:28.653573 32207 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35116:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 02:19:28.654381 32207 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:19:28.663266 32465 tablet_service.cc:1467] Processing CreateTablet for tablet 892ece785cc34f98a448bcac42c54788 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fe33a9ba0459484aabdfc05149cc3d53]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:28.663254 32333 tablet_service.cc:1467] Processing CreateTablet for tablet 892ece785cc34f98a448bcac42c54788 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fe33a9ba0459484aabdfc05149cc3d53]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:28.663762 32465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892ece785cc34f98a448bcac42c54788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:28.663765 32333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892ece785cc34f98a448bcac42c54788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:28.663668 32597 tablet_service.cc:1467] Processing CreateTablet for tablet 892ece785cc34f98a448bcac42c54788 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fe33a9ba0459484aabdfc05149cc3d53]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:28.664079 32597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892ece785cc34f98a448bcac42c54788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:28.666504 32596 tablet_service.cc:1467] Processing CreateTablet for tablet 1660f3cda0a74bb28119a96fe2204868 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fe33a9ba0459484aabdfc05149cc3d53]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:28.666977 32596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1660f3cda0a74bb28119a96fe2204868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:28.667469 32466 tablet_service.cc:1467] Processing CreateTablet for tablet 1660f3cda0a74bb28119a96fe2204868 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fe33a9ba0459484aabdfc05149cc3d53]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:28.667850 32466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1660f3cda0a74bb28119a96fe2204868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:28.671133 32467 tablet_service.cc:1467] Processing CreateTablet for tablet 4db26cccb3444e7b89889f928a117c55 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fe33a9ba0459484aabdfc05149cc3d53]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:28.671634 32467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db26cccb3444e7b89889f928a117c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:28.675199 32598 tablet_service.cc:1467] Processing CreateTablet for tablet 4db26cccb3444e7b89889f928a117c55 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fe33a9ba0459484aabdfc05149cc3d53]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:28.675604 32598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db26cccb3444e7b89889f928a117c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:28.675604 32334 tablet_service.cc:1467] Processing CreateTablet for tablet 4db26cccb3444e7b89889f928a117c55 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fe33a9ba0459484aabdfc05149cc3d53]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:28.675984 32334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db26cccb3444e7b89889f928a117c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:28.678176 32335 tablet_service.cc:1467] Processing CreateTablet for tablet 1660f3cda0a74bb28119a96fe2204868 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fe33a9ba0459484aabdfc05149cc3d53]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:19:28.678567 32335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1660f3cda0a74bb28119a96fe2204868. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:28.678714   652 tablet_bootstrap.cc:492] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07: Bootstrap starting.
I20240503 02:19:28.680470   652 tablet_bootstrap.cc:654] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:28.691545   654 tablet_bootstrap.cc:492] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af: Bootstrap starting.
I20240503 02:19:28.694036   654 tablet_bootstrap.cc:654] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:28.696812   652 tablet_bootstrap.cc:492] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07: No bootstrap required, opened a new log
I20240503 02:19:28.696992   652 ts_tablet_manager.cc:1367] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.007s
I20240503 02:19:28.697100   651 tablet_bootstrap.cc:492] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25: Bootstrap starting.
I20240503 02:19:28.697690   652 raft_consensus.cc:348] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.697862   652 raft_consensus.cc:374] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:28.697965   652 raft_consensus.cc:729] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6faa7decfa4da2bceb8a4aff176d07, State: Initialized, Role: FOLLOWER
I20240503 02:19:28.698192   652 consensus_queue.cc:260] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.698469   651 tablet_bootstrap.cc:654] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:28.698891   652 ts_tablet_manager.cc:1398] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:19:28.699263   652 tablet_bootstrap.cc:492] T 1660f3cda0a74bb28119a96fe2204868 P 7f6faa7decfa4da2bceb8a4aff176d07: Bootstrap starting.
I20240503 02:19:28.699941   654 tablet_bootstrap.cc:492] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af: No bootstrap required, opened a new log
I20240503 02:19:28.700132   654 ts_tablet_manager.cc:1367] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I20240503 02:19:28.700973   654 raft_consensus.cc:348] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.701210   654 raft_consensus.cc:374] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:28.701324   654 raft_consensus.cc:729] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Initialized, Role: FOLLOWER
I20240503 02:19:28.701427   652 tablet_bootstrap.cc:654] T 1660f3cda0a74bb28119a96fe2204868 P 7f6faa7decfa4da2bceb8a4aff176d07: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:28.701603   654 consensus_queue.cc:260] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.705201   654 ts_tablet_manager.cc:1398] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:19:28.705808   652 tablet_bootstrap.cc:492] T 1660f3cda0a74bb28119a96fe2204868 P 7f6faa7decfa4da2bceb8a4aff176d07: No bootstrap required, opened a new log
I20240503 02:19:28.706075   652 ts_tablet_manager.cc:1367] T 1660f3cda0a74bb28119a96fe2204868 P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.004s
I20240503 02:19:28.706092   654 tablet_bootstrap.cc:492] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af: Bootstrap starting.
I20240503 02:19:28.707391   652 raft_consensus.cc:348] T 1660f3cda0a74bb28119a96fe2204868 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.707731   652 raft_consensus.cc:374] T 1660f3cda0a74bb28119a96fe2204868 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:28.707785   654 tablet_bootstrap.cc:654] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:28.707834   652 raft_consensus.cc:729] T 1660f3cda0a74bb28119a96fe2204868 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6faa7decfa4da2bceb8a4aff176d07, State: Initialized, Role: FOLLOWER
I20240503 02:19:28.708173   652 consensus_queue.cc:260] T 1660f3cda0a74bb28119a96fe2204868 P 7f6faa7decfa4da2bceb8a4aff176d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.708912   652 ts_tablet_manager.cc:1398] T 1660f3cda0a74bb28119a96fe2204868 P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.000s
I20240503 02:19:28.709365   652 tablet_bootstrap.cc:492] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07: Bootstrap starting.
I20240503 02:19:28.710997   652 tablet_bootstrap.cc:654] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:28.711458   651 tablet_bootstrap.cc:492] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25: No bootstrap required, opened a new log
I20240503 02:19:28.711655   651 ts_tablet_manager.cc:1367] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25: Time spent bootstrapping tablet: real 0.015s	user 0.008s	sys 0.004s
I20240503 02:19:28.712728   651 raft_consensus.cc:348] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.713073   651 raft_consensus.cc:374] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:28.713212   651 raft_consensus.cc:729] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9296c071f334a2c9078772294a56f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:28.713601   651 consensus_queue.cc:260] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.714537   651 ts_tablet_manager.cc:1398] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 02:19:28.715386   651 tablet_bootstrap.cc:492] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25: Bootstrap starting.
I20240503 02:19:28.717671   651 tablet_bootstrap.cc:654] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:28.719750   652 tablet_bootstrap.cc:492] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07: No bootstrap required, opened a new log
I20240503 02:19:28.719820   651 tablet_bootstrap.cc:492] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25: No bootstrap required, opened a new log
I20240503 02:19:28.719950   652 ts_tablet_manager.cc:1367] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent bootstrapping tablet: real 0.011s	user 0.010s	sys 0.000s
I20240503 02:19:28.719985   651 ts_tablet_manager.cc:1367] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.002s
I20240503 02:19:28.720793   651 raft_consensus.cc:348] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.721129   651 raft_consensus.cc:374] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:28.721124   652 raft_consensus.cc:348] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.721257   651 raft_consensus.cc:729] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9296c071f334a2c9078772294a56f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:28.721382   652 raft_consensus.cc:374] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:28.721513   652 raft_consensus.cc:729] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6faa7decfa4da2bceb8a4aff176d07, State: Initialized, Role: FOLLOWER
I20240503 02:19:28.721632   651 consensus_queue.cc:260] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.721776   652 consensus_queue.cc:260] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.722374   651 ts_tablet_manager.cc:1398] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:19:28.722389   652 ts_tablet_manager.cc:1398] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:19:28.722721   651 tablet_bootstrap.cc:492] T 1660f3cda0a74bb28119a96fe2204868 P d9296c071f334a2c9078772294a56f25: Bootstrap starting.
I20240503 02:19:28.723780   654 tablet_bootstrap.cc:492] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af: No bootstrap required, opened a new log
I20240503 02:19:28.723949   654 ts_tablet_manager.cc:1367] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af: Time spent bootstrapping tablet: real 0.018s	user 0.014s	sys 0.000s
I20240503 02:19:28.724418   651 tablet_bootstrap.cc:654] T 1660f3cda0a74bb28119a96fe2204868 P d9296c071f334a2c9078772294a56f25: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:28.724786   654 raft_consensus.cc:348] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.725050   654 raft_consensus.cc:374] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:28.725153   654 raft_consensus.cc:729] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Initialized, Role: FOLLOWER
I20240503 02:19:28.725425   654 consensus_queue.cc:260] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.726323   651 tablet_bootstrap.cc:492] T 1660f3cda0a74bb28119a96fe2204868 P d9296c071f334a2c9078772294a56f25: No bootstrap required, opened a new log
I20240503 02:19:28.726497   651 ts_tablet_manager.cc:1367] T 1660f3cda0a74bb28119a96fe2204868 P d9296c071f334a2c9078772294a56f25: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:19:28.726656   654 ts_tablet_manager.cc:1398] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:19:28.727242   651 raft_consensus.cc:348] T 1660f3cda0a74bb28119a96fe2204868 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.727402   651 raft_consensus.cc:374] T 1660f3cda0a74bb28119a96fe2204868 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:28.727537   651 raft_consensus.cc:729] T 1660f3cda0a74bb28119a96fe2204868 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9296c071f334a2c9078772294a56f25, State: Initialized, Role: FOLLOWER
I20240503 02:19:28.727789   651 consensus_queue.cc:260] T 1660f3cda0a74bb28119a96fe2204868 P d9296c071f334a2c9078772294a56f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.728401   651 ts_tablet_manager.cc:1398] T 1660f3cda0a74bb28119a96fe2204868 P d9296c071f334a2c9078772294a56f25: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:19:28.729236   654 tablet_bootstrap.cc:492] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af: Bootstrap starting.
I20240503 02:19:28.730929   654 tablet_bootstrap.cc:654] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af: Neither blocks nor log segments found. Creating new log.
I20240503 02:19:28.732617   654 tablet_bootstrap.cc:492] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af: No bootstrap required, opened a new log
I20240503 02:19:28.732743   654 ts_tablet_manager.cc:1367] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:19:28.733343   654 raft_consensus.cc:348] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.733500   654 raft_consensus.cc:374] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:19:28.733599   654 raft_consensus.cc:729] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Initialized, Role: FOLLOWER
I20240503 02:19:28.733806   654 consensus_queue.cc:260] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.734267   654 ts_tablet_manager.cc:1398] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:19:28.790402   643 raft_consensus.cc:482] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:28.790581   643 raft_consensus.cc:504] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.791306   643 leader_election.cc:288] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:28.791785 32354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1660f3cda0a74bb28119a96fe2204868" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9296c071f334a2c9078772294a56f25" is_pre_election: true
I20240503 02:19:28.791838 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1660f3cda0a74bb28119a96fe2204868" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" is_pre_election: true
I20240503 02:19:28.792011 32354 raft_consensus.cc:2454] T 1660f3cda0a74bb28119a96fe2204868 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 0.
I20240503 02:19:28.792104 32617 raft_consensus.cc:2454] T 1660f3cda0a74bb28119a96fe2204868 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 0.
I20240503 02:19:28.792443 32420 leader_election.cc:302] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37afeff9fa264bc69d4f70185f9665af, d9296c071f334a2c9078772294a56f25; no voters: 
I20240503 02:19:28.792709   643 raft_consensus.cc:2781] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:28.792824   643 raft_consensus.cc:482] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:28.792915   643 raft_consensus.cc:3037] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:28.794901   643 raft_consensus.cc:504] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.795477   643 leader_election.cc:288] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 election: Requested vote from peers d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:28.795946 32354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1660f3cda0a74bb28119a96fe2204868" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9296c071f334a2c9078772294a56f25"
I20240503 02:19:28.795992 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1660f3cda0a74bb28119a96fe2204868" candidate_uuid: "37afeff9fa264bc69d4f70185f9665af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07"
I20240503 02:19:28.796129 32354 raft_consensus.cc:3037] T 1660f3cda0a74bb28119a96fe2204868 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:28.796254 32617 raft_consensus.cc:3037] T 1660f3cda0a74bb28119a96fe2204868 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:28.797996 32354 raft_consensus.cc:2454] T 1660f3cda0a74bb28119a96fe2204868 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 1.
I20240503 02:19:28.798059 32617 raft_consensus.cc:2454] T 1660f3cda0a74bb28119a96fe2204868 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37afeff9fa264bc69d4f70185f9665af in term 1.
I20240503 02:19:28.798538 32421 leader_election.cc:302] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 37afeff9fa264bc69d4f70185f9665af, 7f6faa7decfa4da2bceb8a4aff176d07, d9296c071f334a2c9078772294a56f25; no voters: 
I20240503 02:19:28.798923   643 raft_consensus.cc:2781] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:28.799067   643 raft_consensus.cc:686] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Becoming Leader. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Running, Role: LEADER
I20240503 02:19:28.799324   643 consensus_queue.cc:237] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.802335 32207 catalog_manager.cc:5484] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af reported cstate change: term changed from 0 to 1, leader changed from <none> to 37afeff9fa264bc69d4f70185f9665af (127.0.145.2). New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:28.850055   509 raft_consensus.cc:482] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:28.850268   509 raft_consensus.cc:504] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.851820   509 leader_election.cc:288] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993), d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943)
I20240503 02:19:28.858440 32354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892ece785cc34f98a448bcac42c54788" candidate_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9296c071f334a2c9078772294a56f25" is_pre_election: true
I20240503 02:19:28.858732 32354 raft_consensus.cc:2454] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f6faa7decfa4da2bceb8a4aff176d07 in term 0.
I20240503 02:19:28.858793 32487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892ece785cc34f98a448bcac42c54788" candidate_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37afeff9fa264bc69d4f70185f9665af" is_pre_election: true
I20240503 02:19:28.859032 32487 raft_consensus.cc:2454] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f6faa7decfa4da2bceb8a4aff176d07 in term 0.
I20240503 02:19:28.859227 32551 leader_election.cc:302] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6faa7decfa4da2bceb8a4aff176d07, d9296c071f334a2c9078772294a56f25; no voters: 
I20240503 02:19:28.859668   509 raft_consensus.cc:2781] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:28.859804   509 raft_consensus.cc:482] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:28.859906   509 raft_consensus.cc:3037] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:28.861796   509 raft_consensus.cc:504] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.862377   509 leader_election.cc:288] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [CANDIDATE]: Term 1 election: Requested vote from peers 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993), d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943)
I20240503 02:19:28.862936 32354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892ece785cc34f98a448bcac42c54788" candidate_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9296c071f334a2c9078772294a56f25"
I20240503 02:19:28.862946 32487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892ece785cc34f98a448bcac42c54788" candidate_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37afeff9fa264bc69d4f70185f9665af"
I20240503 02:19:28.863147 32354 raft_consensus.cc:3037] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:28.863162 32487 raft_consensus.cc:3037] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:28.865006 32354 raft_consensus.cc:2454] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f6faa7decfa4da2bceb8a4aff176d07 in term 1.
I20240503 02:19:28.865012 32487 raft_consensus.cc:2454] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f6faa7decfa4da2bceb8a4aff176d07 in term 1.
I20240503 02:19:28.865440 32552 leader_election.cc:302] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 37afeff9fa264bc69d4f70185f9665af, 7f6faa7decfa4da2bceb8a4aff176d07, d9296c071f334a2c9078772294a56f25; no voters: 
I20240503 02:19:28.865773   509 raft_consensus.cc:2781] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:28.866195   509 raft_consensus.cc:686] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 LEADER]: Becoming Leader. State: Replica: 7f6faa7decfa4da2bceb8a4aff176d07, State: Running, Role: LEADER
I20240503 02:19:28.866739   509 consensus_queue.cc:237] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.870044 32207 catalog_manager.cc:5484] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3). New cstate: current_term: 1 leader_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: HEALTHY } } }
I20240503 02:19:28.893729   620 raft_consensus.cc:482] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:19:28.893949   620 raft_consensus.cc:504] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.894769   620 leader_election.cc:288] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:28.895260 32487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db26cccb3444e7b89889f928a117c55" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37afeff9fa264bc69d4f70185f9665af" is_pre_election: true
I20240503 02:19:28.895303 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db26cccb3444e7b89889f928a117c55" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" is_pre_election: true
I20240503 02:19:28.895488 32487 raft_consensus.cc:2454] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 0.
I20240503 02:19:28.895510 32617 raft_consensus.cc:2454] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 0.
I20240503 02:19:28.895927 32289 leader_election.cc:302] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6faa7decfa4da2bceb8a4aff176d07, d9296c071f334a2c9078772294a56f25; no voters: 
I20240503 02:19:28.896190   620 raft_consensus.cc:2781] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:19:28.896317   620 raft_consensus.cc:482] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:19:28.896420   620 raft_consensus.cc:3037] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:28.898396   620 raft_consensus.cc:504] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.898975   620 leader_election.cc:288] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [CANDIDATE]: Term 1 election: Requested vote from peers 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:28.899479 32487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db26cccb3444e7b89889f928a117c55" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37afeff9fa264bc69d4f70185f9665af"
I20240503 02:19:28.899504 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db26cccb3444e7b89889f928a117c55" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07"
I20240503 02:19:28.899695 32617 raft_consensus.cc:3037] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:28.899717 32487 raft_consensus.cc:3037] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:19:28.902642 32487 raft_consensus.cc:2454] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 1.
I20240503 02:19:28.902660 32617 raft_consensus.cc:2454] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 1.
I20240503 02:19:28.903100 32289 leader_election.cc:302] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6faa7decfa4da2bceb8a4aff176d07, d9296c071f334a2c9078772294a56f25; no voters: 
I20240503 02:19:28.903391   620 raft_consensus.cc:2781] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:19:28.903546   620 raft_consensus.cc:686] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [term 1 LEADER]: Becoming Leader. State: Replica: d9296c071f334a2c9078772294a56f25, State: Running, Role: LEADER
I20240503 02:19:28.903860   620 consensus_queue.cc:237] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } }
I20240503 02:19:28.906641 32207 catalog_manager.cc:5484] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9296c071f334a2c9078772294a56f25 (127.0.145.1). New cstate: current_term: 1 leader_uuid: "d9296c071f334a2c9078772294a56f25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: UNKNOWN } } }
W20240503 02:19:29.272420   669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:29.272725   669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:29.280259   669 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 02:19:29.328297   646 consensus_queue.cc:1035] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:29.337109   693 consensus_queue.cc:1035] T 1660f3cda0a74bb28119a96fe2204868 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:29.384259   620 consensus_queue.cc:1035] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:29.389781   506 consensus_queue.cc:1035] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:29.437420   509 consensus_queue.cc:1035] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:19:29.439563   509 consensus_queue.cc:1035] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 264.848 ms
  time per row: 0.132424 ms
I20240503 02:19:29.706816   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.5:41015
--local_ip_for_outbound_sockets=127.0.145.5
--tserver_master_addrs=127.0.145.62:35797
--webserver_port=39607
--webserver_interface=127.0.145.5
--builtin_ntp_servers=127.0.145.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:19:30.051544   734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:30.051865   734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:19:30.052008   734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:30.060050   734 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:19:30.060210   734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:19:30.060364   734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.5
I20240503 02:19:30.069355   734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.5:41015
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.145.5
--webserver_port=39607
--tserver_master_addrs=127.0.145.62:35797
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.5
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:19:30.070726   734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:19:30.072685   734 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:19:30.085116   739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:30.085691   740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:30.086359   742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:19:30.086946   734 server_base.cc:1034] running on GCE node
I20240503 02:19:30.087551   734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:19:30.088546   734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:19:30.089733   734 hybrid_clock.cc:648] HybridClock initialized: now 1714702770089691 us; error 36 us; skew 500 ppm
I20240503 02:19:30.090045   734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:19:30.092311   734 webserver.cc:460] Webserver started at http://127.0.145.5:39607/ using document root <none> and password file <none>
I20240503 02:19:30.093035   734 fs_manager.cc:359] Metadata directory not provided
I20240503 02:19:30.093187   734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:19:30.099118   734 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.001s
I20240503 02:19:30.102264   748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:30.103920   734 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:19:30.104116   734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8c3494a1fbad4d9aa0aa551169b78fa9"
format_stamp: "Formatted at 2024-05-03 02:19:22 on dist-test-slave-gq9h"
I20240503 02:19:30.105180   734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:19:30.138494   734 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:19:30.139372   734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:19:30.139655   734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:19:30.140599   734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:19:30.142117   734 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:19:30.142187   734 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:30.142310   734 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:19:30.142371   734 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:30.172827   734 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.5:41015
I20240503 02:19:30.172871   860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.5:41015 every 8 connection(s)
I20240503 02:19:30.174337   734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:19:30.177441   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 734
I20240503 02:19:30.178303   580 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
/tmp/dist-test-taskfpuZdt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.145.4:37397
--local_ip_for_outbound_sockets=127.0.145.4
--tserver_master_addrs=127.0.145.62:35797
--webserver_port=35309
--webserver_interface=127.0.145.4
--builtin_ntp_servers=127.0.145.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:19:30.188673   861 heartbeater.cc:347] Connected to a master server at 127.0.145.62:35797
I20240503 02:19:30.189024   861 heartbeater.cc:464] Registering TS with master...
I20240503 02:19:30.190080   861 heartbeater.cc:511] Master 127.0.145.62:35797 requested a full tablet report, sending...
I20240503 02:19:30.192062 32213 ts_manager.cc:194] Re-registered known tserver with Master: 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5:41015)
I20240503 02:19:30.193349 32213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.5:39451
W20240503 02:19:30.530337   865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:19:30.530680   865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:19:30.530793   865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:19:30.538973   865 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:19:30.539135   865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:19:30.539343   865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.145.4
I20240503 02:19:30.548566   865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.145.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.145.4:37397
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.145.4
--webserver_port=35309
--tserver_master_addrs=127.0.145.62:35797
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.145.4
--log_dir=/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:19:30.549892   865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:19:30.551677   865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:19:30.564491   871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:30.564848   872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:19:30.565323   874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:19:30.568856   865 server_base.cc:1034] running on GCE node
I20240503 02:19:30.569561   865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:19:30.570577   865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:19:30.571856   865 hybrid_clock.cc:648] HybridClock initialized: now 1714702770571789 us; error 59 us; skew 500 ppm
I20240503 02:19:30.572208   865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:19:30.575033   865 webserver.cc:460] Webserver started at http://127.0.145.4:35309/ using document root <none> and password file <none>
I20240503 02:19:30.575805   865 fs_manager.cc:359] Metadata directory not provided
I20240503 02:19:30.575953   865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:19:30.581504   865 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240503 02:19:30.584426   880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:30.585865   865 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240503 02:19:30.586030   865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b5704d091d784c849a96fe720e53cc45"
format_stamp: "Formatted at 2024-05-03 02:19:21 on dist-test-slave-gq9h"
I20240503 02:19:30.586944   865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:19:30.627956   865 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:19:30.628799   865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:19:30.629073   865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:19:30.630055   865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:19:30.631654   865 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:19:30.631731   865 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:30.631855   865 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:19:30.631908   865 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:19:30.664579   865 rpc_server.cc:307] RPC server started. Bound to: 127.0.145.4:37397
I20240503 02:19:30.664595   992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.145.4:37397 every 8 connection(s)
I20240503 02:19:30.666070   865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:19:30.673272   580 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu as pid 865
I20240503 02:19:30.679360   993 heartbeater.cc:347] Connected to a master server at 127.0.145.62:35797
I20240503 02:19:30.679718   993 heartbeater.cc:464] Registering TS with master...
I20240503 02:19:30.680771   993 heartbeater.cc:511] Master 127.0.145.62:35797 requested a full tablet report, sending...
I20240503 02:19:30.682509 32213 ts_manager.cc:194] Re-registered known tserver with Master: b5704d091d784c849a96fe720e53cc45 (127.0.145.4:37397)
I20240503 02:19:30.683331 32213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.145.4:49535
I20240503 02:19:31.180029   796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:31.185629 32467 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:31.195864 32598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:31.196723   861 heartbeater.cc:503] Master 127.0.145.62:35797 was elected leader, sending a full tablet report...
I20240503 02:19:31.209415   928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:31.246024 32333 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:31.394353 32467 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:31.396891 32598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:31.406137   928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:31.410717 32333 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:31.416605   796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:31.610018   796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:31.622443   928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:31.623503 32598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:31.624843 32467 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:31.628494 32333 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:31.685834   993 heartbeater.cc:503] Master 127.0.145.62:35797 was elected leader, sending a full tablet report...
I20240503 02:19:31.785579 32467 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:31.786096 32598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:31.798082   928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:31.806633   796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:31.808403 32333 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:31.879621 32355 consensus_queue.cc:237] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:31.882313 32486 raft_consensus.cc:1261] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:31.882313 32618 raft_consensus.cc:1261] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:31.883050  1138 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:31.883535  1137 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:31.886935  1138 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
W20240503 02:19:31.889683 32291 consensus_peers.cc:485] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 -> Peer 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5:41015): Couldn't send request to peer 8c3494a1fbad4d9aa0aa551169b78fa9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e1f43f03ccf472c821943d848064d63. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:31.891160 32486 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:31.892127 32208 catalog_manager.cc:5484] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 reported cstate change: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "d9296c071f334a2c9078772294a56f25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:31.893942 32618 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:31.912711 32486 consensus_queue.cc:237] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:31.915565 32355 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:31.915805 32618 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:31.916296   997 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:31.916652  1152 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:19:31.927326 32423 consensus_peers.cc:485] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af -> Peer 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5:41015): Couldn't send request to peer 8c3494a1fbad4d9aa0aa551169b78fa9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10541b8e86304eebbe61b7ebc176eb9d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:31.927471  1152 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:31.927845 32355 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:31.932032 32618 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:31.933899 32212 catalog_manager.cc:5484] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 reported cstate change: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:31.947800 32618 consensus_queue.cc:237] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:31.951637 32486 raft_consensus.cc:1261] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Refusing update from remote peer 7f6faa7decfa4da2bceb8a4aff176d07: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:31.953248   866 consensus_queue.cc:1035] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:31.955792 32355 raft_consensus.cc:1261] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Refusing update from remote peer 7f6faa7decfa4da2bceb8a4aff176d07: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:31.956387   866 consensus_queue.cc:1035] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:31.957063   866 raft_consensus.cc:2932] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
W20240503 02:19:31.963562 32554 consensus_peers.cc:485] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 -> Peer 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5:41015): Couldn't send request to peer 8c3494a1fbad4d9aa0aa551169b78fa9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 892ece785cc34f98a448bcac42c54788. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:31.965484 32208 catalog_manager.cc:5484] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 reported cstate change: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:31.972266 32354 raft_consensus.cc:2932] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:31.980741 32486 raft_consensus.cc:2932] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:31.994907 32355 consensus_queue.cc:237] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:31.997081 32618 raft_consensus.cc:1261] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:31.997121 32486 raft_consensus.cc:1261] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:19:31.997524 32291 consensus_peers.cc:485] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 -> Peer 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5:41015): Couldn't send request to peer 8c3494a1fbad4d9aa0aa551169b78fa9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4db26cccb3444e7b89889f928a117c55. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:31.997674   620 consensus_queue.cc:1035] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:31.997977  1138 consensus_queue.cc:1035] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:32.000679  1138 raft_consensus.cc:2932] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:32.001451 32618 raft_consensus.cc:2932] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:32.001498 32486 raft_consensus.cc:2932] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:32.006318 32208 catalog_manager.cc:5484] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 reported cstate change: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "d9296c071f334a2c9078772294a56f25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:32.022034 32486 consensus_queue.cc:237] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:32.024302 32618 raft_consensus.cc:1261] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:32.024394 32355 raft_consensus.cc:1261] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:19:32.024468 32423 consensus_peers.cc:485] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af -> Peer 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5:41015): Couldn't send request to peer 8c3494a1fbad4d9aa0aa551169b78fa9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d90f1fe288904e879c20a7e9c9d7a140. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:32.024933  1152 consensus_queue.cc:1035] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:32.025215  1153 consensus_queue.cc:1035] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:32.027886   997 raft_consensus.cc:2932] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:32.028506 32355 raft_consensus.cc:2932] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:32.028560 32618 raft_consensus.cc:2932] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:32.034044 32213 catalog_manager.cc:5484] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07 reported cstate change: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:32.303563  1222 ts_tablet_manager.cc:902] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9: Initiating tablet copy from peer d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943)
I20240503 02:19:32.311406  1222 tablet_copy_client.cc:308] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:45943
I20240503 02:19:32.318449 32375 tablet_copy_service.cc:140] P d9296c071f334a2c9078772294a56f25: Received BeginTabletCopySession request for tablet 1e1f43f03ccf472c821943d848064d63 from peer 8c3494a1fbad4d9aa0aa551169b78fa9 ({username='slave'} at 127.0.145.5:54547)
I20240503 02:19:32.318759 32375 tablet_copy_service.cc:161] P d9296c071f334a2c9078772294a56f25: Beginning new tablet copy session on tablet 1e1f43f03ccf472c821943d848064d63 from peer 8c3494a1fbad4d9aa0aa551169b78fa9 at {username='slave'} at 127.0.145.5:54547: session id = 8c3494a1fbad4d9aa0aa551169b78fa9-1e1f43f03ccf472c821943d848064d63
I20240503 02:19:32.321400 32375 tablet_copy_source_session.cc:205] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:32.326503  1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1f43f03ccf472c821943d848064d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:32.338922  1222 tablet_copy_client.cc:783] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:32.339413  1222 tablet_copy_client.cc:647] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:32.342260  1222 tablet_copy_client.cc:523] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:32.345391  1222 tablet_bootstrap.cc:492] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap starting.
I20240503 02:19:32.354720  1222 log.cc:824] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9: Log is configured to *not* fsync() on all Append() calls
I20240503 02:19:32.374342  1244 ts_tablet_manager.cc:902] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9: Initiating tablet copy from peer 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:32.376026  1244 tablet_copy_client.cc:308] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.3:38283
I20240503 02:19:32.383263 32638 tablet_copy_service.cc:140] P 7f6faa7decfa4da2bceb8a4aff176d07: Received BeginTabletCopySession request for tablet 892ece785cc34f98a448bcac42c54788 from peer 8c3494a1fbad4d9aa0aa551169b78fa9 ({username='slave'} at 127.0.145.5:33455)
I20240503 02:19:32.383499 32638 tablet_copy_service.cc:161] P 7f6faa7decfa4da2bceb8a4aff176d07: Beginning new tablet copy session on tablet 892ece785cc34f98a448bcac42c54788 from peer 8c3494a1fbad4d9aa0aa551169b78fa9 at {username='slave'} at 127.0.145.5:33455: session id = 8c3494a1fbad4d9aa0aa551169b78fa9-892ece785cc34f98a448bcac42c54788
I20240503 02:19:32.386143 32638 tablet_copy_source_session.cc:205] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:32.387985  1244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892ece785cc34f98a448bcac42c54788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:32.389926  1222 tablet_bootstrap.cc:492] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:32.390403  1222 tablet_bootstrap.cc:492] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap complete.
I20240503 02:19:32.390918  1222 ts_tablet_manager.cc:1367] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9: Time spent bootstrapping tablet: real 0.046s	user 0.039s	sys 0.008s
I20240503 02:19:32.392691  1244 tablet_copy_client.cc:783] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:32.393014  1244 tablet_copy_client.cc:647] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:32.395170  1244 tablet_copy_client.cc:523] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:32.396168  1222 raft_consensus.cc:348] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:32.396572  1222 raft_consensus.cc:729] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8c3494a1fbad4d9aa0aa551169b78fa9, State: Initialized, Role: LEARNER
I20240503 02:19:32.397101  1222 consensus_queue.cc:260] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:32.398025  1244 tablet_bootstrap.cc:492] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap starting.
I20240503 02:19:32.398939  1222 ts_tablet_manager.cc:1398] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20240503 02:19:32.399955 32375 tablet_copy_service.cc:342] P d9296c071f334a2c9078772294a56f25: Request end of tablet copy session 8c3494a1fbad4d9aa0aa551169b78fa9-1e1f43f03ccf472c821943d848064d63 received from {username='slave'} at 127.0.145.5:54547
I20240503 02:19:32.400146 32375 tablet_copy_service.cc:433] P d9296c071f334a2c9078772294a56f25: ending tablet copy session 8c3494a1fbad4d9aa0aa551169b78fa9-1e1f43f03ccf472c821943d848064d63 on tablet 1e1f43f03ccf472c821943d848064d63 with peer 8c3494a1fbad4d9aa0aa551169b78fa9
I20240503 02:19:32.419050  1222 ts_tablet_manager.cc:902] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9: Initiating tablet copy from peer 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993)
I20240503 02:19:32.420375  1222 tablet_copy_client.cc:308] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:41993
I20240503 02:19:32.427299 32507 tablet_copy_service.cc:140] P 37afeff9fa264bc69d4f70185f9665af: Received BeginTabletCopySession request for tablet 10541b8e86304eebbe61b7ebc176eb9d from peer 8c3494a1fbad4d9aa0aa551169b78fa9 ({username='slave'} at 127.0.145.5:36803)
I20240503 02:19:32.427570 32507 tablet_copy_service.cc:161] P 37afeff9fa264bc69d4f70185f9665af: Beginning new tablet copy session on tablet 10541b8e86304eebbe61b7ebc176eb9d from peer 8c3494a1fbad4d9aa0aa551169b78fa9 at {username='slave'} at 127.0.145.5:36803: session id = 8c3494a1fbad4d9aa0aa551169b78fa9-10541b8e86304eebbe61b7ebc176eb9d
I20240503 02:19:32.431313 32507 tablet_copy_source_session.cc:205] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:32.433053  1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10541b8e86304eebbe61b7ebc176eb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:32.434298  1244 tablet_bootstrap.cc:492] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:32.434800  1244 tablet_bootstrap.cc:492] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap complete.
I20240503 02:19:32.435048  1244 ts_tablet_manager.cc:1367] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9: Time spent bootstrapping tablet: real 0.037s	user 0.027s	sys 0.011s
I20240503 02:19:32.435947  1244 raft_consensus.cc:348] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:32.436164  1244 raft_consensus.cc:729] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8c3494a1fbad4d9aa0aa551169b78fa9, State: Initialized, Role: LEARNER
I20240503 02:19:32.436400  1244 consensus_queue.cc:260] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:32.436918  1244 ts_tablet_manager.cc:1398] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:19:32.437907 32638 tablet_copy_service.cc:342] P 7f6faa7decfa4da2bceb8a4aff176d07: Request end of tablet copy session 8c3494a1fbad4d9aa0aa551169b78fa9-892ece785cc34f98a448bcac42c54788 received from {username='slave'} at 127.0.145.5:33455
I20240503 02:19:32.438094 32638 tablet_copy_service.cc:433] P 7f6faa7decfa4da2bceb8a4aff176d07: ending tablet copy session 8c3494a1fbad4d9aa0aa551169b78fa9-892ece785cc34f98a448bcac42c54788 on tablet 892ece785cc34f98a448bcac42c54788 with peer 8c3494a1fbad4d9aa0aa551169b78fa9
I20240503 02:19:32.438350  1222 tablet_copy_client.cc:783] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:32.438673  1222 tablet_copy_client.cc:647] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:32.440925  1222 tablet_copy_client.cc:523] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:32.443251  1222 tablet_bootstrap.cc:492] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap starting.
I20240503 02:19:32.470384  1222 tablet_bootstrap.cc:492] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:32.470829  1222 tablet_bootstrap.cc:492] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap complete.
I20240503 02:19:32.471078  1222 ts_tablet_manager.cc:1367] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9: Time spent bootstrapping tablet: real 0.028s	user 0.024s	sys 0.004s
I20240503 02:19:32.471707  1222 raft_consensus.cc:348] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:32.471915  1222 raft_consensus.cc:729] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8c3494a1fbad4d9aa0aa551169b78fa9, State: Initialized, Role: LEARNER
I20240503 02:19:32.472121  1222 consensus_queue.cc:260] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:32.472627  1222 ts_tablet_manager.cc:1398] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:19:32.473646 32507 tablet_copy_service.cc:342] P 37afeff9fa264bc69d4f70185f9665af: Request end of tablet copy session 8c3494a1fbad4d9aa0aa551169b78fa9-10541b8e86304eebbe61b7ebc176eb9d received from {username='slave'} at 127.0.145.5:36803
I20240503 02:19:32.473829 32507 tablet_copy_service.cc:433] P 37afeff9fa264bc69d4f70185f9665af: ending tablet copy session 8c3494a1fbad4d9aa0aa551169b78fa9-10541b8e86304eebbe61b7ebc176eb9d on tablet 10541b8e86304eebbe61b7ebc176eb9d with peer 8c3494a1fbad4d9aa0aa551169b78fa9
I20240503 02:19:32.570346  1222 ts_tablet_manager.cc:902] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9: Initiating tablet copy from peer d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943)
I20240503 02:19:32.577579  1222 tablet_copy_client.cc:308] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:45943
I20240503 02:19:32.578366 32375 tablet_copy_service.cc:140] P d9296c071f334a2c9078772294a56f25: Received BeginTabletCopySession request for tablet 4db26cccb3444e7b89889f928a117c55 from peer 8c3494a1fbad4d9aa0aa551169b78fa9 ({username='slave'} at 127.0.145.5:54547)
I20240503 02:19:32.578562 32375 tablet_copy_service.cc:161] P d9296c071f334a2c9078772294a56f25: Beginning new tablet copy session on tablet 4db26cccb3444e7b89889f928a117c55 from peer 8c3494a1fbad4d9aa0aa551169b78fa9 at {username='slave'} at 127.0.145.5:54547: session id = 8c3494a1fbad4d9aa0aa551169b78fa9-4db26cccb3444e7b89889f928a117c55
I20240503 02:19:32.581594 32375 tablet_copy_source_session.cc:205] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:32.582924  1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db26cccb3444e7b89889f928a117c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:32.585208  1244 ts_tablet_manager.cc:902] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9: Initiating tablet copy from peer 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993)
I20240503 02:19:32.589740  1222 tablet_copy_client.cc:783] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:32.590057  1222 tablet_copy_client.cc:647] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:32.592309  1222 tablet_copy_client.cc:523] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:32.593139  1244 tablet_copy_client.cc:308] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:41993
I20240503 02:19:32.593878 32507 tablet_copy_service.cc:140] P 37afeff9fa264bc69d4f70185f9665af: Received BeginTabletCopySession request for tablet d90f1fe288904e879c20a7e9c9d7a140 from peer 8c3494a1fbad4d9aa0aa551169b78fa9 ({username='slave'} at 127.0.145.5:36803)
I20240503 02:19:32.594067 32507 tablet_copy_service.cc:161] P 37afeff9fa264bc69d4f70185f9665af: Beginning new tablet copy session on tablet d90f1fe288904e879c20a7e9c9d7a140 from peer 8c3494a1fbad4d9aa0aa551169b78fa9 at {username='slave'} at 127.0.145.5:36803: session id = 8c3494a1fbad4d9aa0aa551169b78fa9-d90f1fe288904e879c20a7e9c9d7a140
I20240503 02:19:32.595800  1222 tablet_bootstrap.cc:492] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap starting.
I20240503 02:19:32.596433 32507 tablet_copy_source_session.cc:205] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:32.597683  1244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90f1fe288904e879c20a7e9c9d7a140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:32.604344  1244 tablet_copy_client.cc:783] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:32.604631  1244 tablet_copy_client.cc:647] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:32.608069  1244 tablet_copy_client.cc:523] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:32.612753  1244 tablet_bootstrap.cc:492] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap starting.
I20240503 02:19:32.647814  1222 tablet_bootstrap.cc:492] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:32.648260  1222 tablet_bootstrap.cc:492] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap complete.
I20240503 02:19:32.648522  1222 ts_tablet_manager.cc:1367] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9: Time spent bootstrapping tablet: real 0.053s	user 0.035s	sys 0.013s
I20240503 02:19:32.649230  1222 raft_consensus.cc:348] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:32.649434  1222 raft_consensus.cc:729] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8c3494a1fbad4d9aa0aa551169b78fa9, State: Initialized, Role: LEARNER
I20240503 02:19:32.649641  1222 consensus_queue.cc:260] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:32.650198  1222 ts_tablet_manager.cc:1398] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:19:32.651113 32375 tablet_copy_service.cc:342] P d9296c071f334a2c9078772294a56f25: Request end of tablet copy session 8c3494a1fbad4d9aa0aa551169b78fa9-4db26cccb3444e7b89889f928a117c55 received from {username='slave'} at 127.0.145.5:54547
I20240503 02:19:32.651247 32375 tablet_copy_service.cc:433] P d9296c071f334a2c9078772294a56f25: ending tablet copy session 8c3494a1fbad4d9aa0aa551169b78fa9-4db26cccb3444e7b89889f928a117c55 on tablet 4db26cccb3444e7b89889f928a117c55 with peer 8c3494a1fbad4d9aa0aa551169b78fa9
I20240503 02:19:32.666266  1244 tablet_bootstrap.cc:492] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:32.666757  1244 tablet_bootstrap.cc:492] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap complete.
I20240503 02:19:32.667006  1244 ts_tablet_manager.cc:1367] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9: Time spent bootstrapping tablet: real 0.054s	user 0.045s	sys 0.007s
I20240503 02:19:32.667663  1244 raft_consensus.cc:348] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:32.667922  1244 raft_consensus.cc:729] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8c3494a1fbad4d9aa0aa551169b78fa9, State: Initialized, Role: LEARNER
I20240503 02:19:32.668138  1244 consensus_queue.cc:260] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:32.668633  1244 ts_tablet_manager.cc:1398] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:19:32.669517 32507 tablet_copy_service.cc:342] P 37afeff9fa264bc69d4f70185f9665af: Request end of tablet copy session 8c3494a1fbad4d9aa0aa551169b78fa9-d90f1fe288904e879c20a7e9c9d7a140 received from {username='slave'} at 127.0.145.5:36803
I20240503 02:19:32.669657 32507 tablet_copy_service.cc:433] P 37afeff9fa264bc69d4f70185f9665af: ending tablet copy session 8c3494a1fbad4d9aa0aa551169b78fa9-d90f1fe288904e879c20a7e9c9d7a140 on tablet d90f1fe288904e879c20a7e9c9d7a140 with peer 8c3494a1fbad4d9aa0aa551169b78fa9
I20240503 02:19:32.695955   816 raft_consensus.cc:1203] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:32.947151   816 raft_consensus.cc:1203] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:32.989013   816 raft_consensus.cc:1203] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:33.096745   816 raft_consensus.cc:1203] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:33.104003  1292 raft_consensus.cc:1050] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25: attempting to promote NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 to VOTER
I20240503 02:19:33.105000  1292 consensus_queue.cc:237] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } }
I20240503 02:19:33.107620 32486 raft_consensus.cc:1261] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.107620 32618 raft_consensus.cc:1261] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.107939   816 raft_consensus.cc:1261] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.108289  1292 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.108733  1328 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.109638  1292 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.112607   620 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.113320   816 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.116576 32618 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.123133 32208 catalog_manager.cc:5484] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 reported cstate change: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d9296c071f334a2c9078772294a56f25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.127530   816 raft_consensus.cc:1203] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:33.130199 32487 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.138684 32355 consensus_queue.cc:237] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } }
I20240503 02:19:33.140944   816 raft_consensus.cc:1261] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:33.141043 32618 raft_consensus.cc:1261] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:33.141698   620 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:33.142329  1292 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:33.144804   620 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.145927   816 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.148263 32198 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1e1f43f03ccf472c821943d848064d63 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:33.149009 32618 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.150684 32208 catalog_manager.cc:5484] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 reported cstate change: config changed from index 5 to 6, VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "d9296c071f334a2c9078772294a56f25" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:33.157516 32467 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e1f43f03ccf472c821943d848064d63 with delete_type TABLET_DATA_TOMBSTONED (TS 37afeff9fa264bc69d4f70185f9665af not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58692
I20240503 02:19:33.158504  1342 tablet_replica.cc:330] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: stopping tablet replica
I20240503 02:19:33.159081  1342 raft_consensus.cc:2229] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:33.159606  1342 raft_consensus.cc:2258] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:33.161042  1342 ts_tablet_manager.cc:1875] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:33.167552  1342 ts_tablet_manager.cc:1888] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:19:33.167732  1342 log.cc:1196] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/1e1f43f03ccf472c821943d848064d63
I20240503 02:19:33.168774 32199 catalog_manager.cc:4830] TS 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993): tablet 1e1f43f03ccf472c821943d848064d63 (table rebalance_test_table_1 [id=71ee030fc6ed4819be1bd2168142c3a4]) successfully deleted
I20240503 02:19:33.448649 32487 consensus_queue.cc:237] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:33.450878 32355 raft_consensus.cc:1261] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:19:33.450966 32618 raft_consensus.cc:1261] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:19:33.451079 32423 consensus_peers.cc:485] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af -> Peer 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5:41015): Couldn't send request to peer 8c3494a1fbad4d9aa0aa551169b78fa9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 991ee5bce8494e908449e49097e14e45. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:33.451546  1186 consensus_queue.cc:1035] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:33.451864  1321 consensus_queue.cc:1035] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:19:33.454427  1278 raft_consensus.cc:2932] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:33.455056 32618 raft_consensus.cc:2932] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:33.455204 32355 raft_consensus.cc:2932] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:33.460147 32208 catalog_manager.cc:5484] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07 reported cstate change: config changed from index -1 to 4, NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) added. New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } } }
I20240503 02:19:33.472232  1392 raft_consensus.cc:1050] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: attempting to promote NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 to VOTER
I20240503 02:19:33.474939  1392 consensus_queue.cc:237] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } }
I20240503 02:19:33.480887 32618 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.480921   816 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.481427 32355 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.481868  1393 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.482357  1186 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.482693  1278 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.486531  1393 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.487401   620 raft_consensus.cc:1050] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25: attempting to promote NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 to VOTER
I20240503 02:19:33.488338   620 consensus_queue.cc:237] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } }
I20240503 02:19:33.490952   815 raft_consensus.cc:1261] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.490928 32355 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.491604   620 consensus_queue.cc:1035] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.491896 32618 raft_consensus.cc:1261] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.492645  1292 consensus_queue.cc:1035] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.493014 32487 raft_consensus.cc:1261] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.494262  1292 consensus_queue.cc:1035] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.496723 32617 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.500643 32208 catalog_manager.cc:5484] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 reported cstate change: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.505299   620 raft_consensus.cc:2932] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.510438 32487 raft_consensus.cc:2932] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.513136   816 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.515868 32486 consensus_queue.cc:237] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } }
I20240503 02:19:33.515671 32617 raft_consensus.cc:2932] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.518762 32355 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:33.519402  1186 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:33.519942   814 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:33.506201   815 raft_consensus.cc:2932] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.523361  1186 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:33.525265  1393 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.525972 32355 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.526389 32211 catalog_manager.cc:5484] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 reported cstate change: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d9296c071f334a2c9078772294a56f25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:33.526917   814 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.529171 32199 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 10541b8e86304eebbe61b7ebc176eb9d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:33.532069 32212 catalog_manager.cc:5484] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 reported cstate change: config changed from index 5 to 6, VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.533128 32355 consensus_queue.cc:237] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } }
I20240503 02:19:33.535181   815 raft_consensus.cc:1261] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:33.535797 32617 raft_consensus.cc:1261] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:33.536782   620 consensus_queue.cc:1035] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:33.537503   620 consensus_queue.cc:1035] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:33.538527 32598 tablet_service.cc:1514] Processing DeleteTablet for tablet 10541b8e86304eebbe61b7ebc176eb9d with delete_type TABLET_DATA_TOMBSTONED (TS 7f6faa7decfa4da2bceb8a4aff176d07 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39876
I20240503 02:19:33.540875   620 raft_consensus.cc:2932] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.542913   815 raft_consensus.cc:2932] T 4db26cccb3444e7b89889f928a117c55 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.544867 32198 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4db26cccb3444e7b89889f928a117c55 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:33.545697 32617 raft_consensus.cc:2932] T 4db26cccb3444e7b89889f928a117c55 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.546645 32211 catalog_manager.cc:5484] T 4db26cccb3444e7b89889f928a117c55 P d9296c071f334a2c9078772294a56f25 reported cstate change: config changed from index 5 to 6, VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) evicted. New cstate: current_term: 1 leader_uuid: "d9296c071f334a2c9078772294a56f25" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:33.551780 32467 tablet_service.cc:1514] Processing DeleteTablet for tablet 4db26cccb3444e7b89889f928a117c55 with delete_type TABLET_DATA_TOMBSTONED (TS 37afeff9fa264bc69d4f70185f9665af not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58692
I20240503 02:19:33.553043  1342 tablet_replica.cc:330] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af: stopping tablet replica
I20240503 02:19:33.553373  1342 raft_consensus.cc:2229] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:33.553725  1342 raft_consensus.cc:2258] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:33.555450  1342 ts_tablet_manager.cc:1875] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:33.567173  1342 ts_tablet_manager.cc:1888] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:19:33.567333  1342 log.cc:1196] T 4db26cccb3444e7b89889f928a117c55 P 37afeff9fa264bc69d4f70185f9665af: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/4db26cccb3444e7b89889f928a117c55
I20240503 02:19:33.568368 32199 catalog_manager.cc:4830] TS 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993): tablet 4db26cccb3444e7b89889f928a117c55 (table rebalance_test_table_2 [id=fe33a9ba0459484aabdfc05149cc3d53]) successfully deleted
I20240503 02:19:33.570825  1412 tablet_replica.cc:330] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: stopping tablet replica
I20240503 02:19:33.571646  1412 raft_consensus.cc:2229] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:33.572182  1412 raft_consensus.cc:2258] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:33.573148  1413 raft_consensus.cc:1050] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07: attempting to promote NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 to VOTER
I20240503 02:19:33.574291  1412 ts_tablet_manager.cc:1875] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:33.575505  1413 consensus_queue.cc:237] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } }
I20240503 02:19:33.583962  1412 ts_tablet_manager.cc:1888] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:19:33.584175  1412 log.cc:1196] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/10541b8e86304eebbe61b7ebc176eb9d
I20240503 02:19:33.585435 32355 raft_consensus.cc:1261] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Refusing update from remote peer 7f6faa7decfa4da2bceb8a4aff176d07: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.585866 32199 catalog_manager.cc:4830] TS 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283): tablet 10541b8e86304eebbe61b7ebc176eb9d (table rebalance_test_table_0 [id=3117cfc63bf2489389e6759839a19b7b]) successfully deleted
I20240503 02:19:33.585886   815 raft_consensus.cc:1261] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Refusing update from remote peer 7f6faa7decfa4da2bceb8a4aff176d07: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.585999 32487 raft_consensus.cc:1261] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Refusing update from remote peer 7f6faa7decfa4da2bceb8a4aff176d07: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.586571  1414 consensus_queue.cc:1035] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.586973  1320 consensus_queue.cc:1035] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.587875  1320 consensus_queue.cc:1035] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.590863  1413 raft_consensus.cc:2932] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.591452 32487 raft_consensus.cc:2932] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.592409 32355 raft_consensus.cc:2932] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.596868 32212 catalog_manager.cc:5484] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af reported cstate change: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.597920   815 raft_consensus.cc:2932] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.611047 32617 consensus_queue.cc:237] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } }
I20240503 02:19:33.613560 32487 raft_consensus.cc:1261] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Refusing update from remote peer 7f6faa7decfa4da2bceb8a4aff176d07: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:33.613699   815 raft_consensus.cc:1261] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Refusing update from remote peer 7f6faa7decfa4da2bceb8a4aff176d07: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:33.614208  1413 consensus_queue.cc:1035] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:33.614869  1320 consensus_queue.cc:1035] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:33.616497  1413 raft_consensus.cc:2932] T 892ece785cc34f98a448bcac42c54788 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d9296c071f334a2c9078772294a56f25 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.617198   815 raft_consensus.cc:2932] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d9296c071f334a2c9078772294a56f25 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.618680 32487 raft_consensus.cc:2932] T 892ece785cc34f98a448bcac42c54788 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d9296c071f334a2c9078772294a56f25 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.622009 32199 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 892ece785cc34f98a448bcac42c54788 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:33.623034 32208 catalog_manager.cc:5484] T 892ece785cc34f98a448bcac42c54788 P 8c3494a1fbad4d9aa0aa551169b78fa9 reported cstate change: config changed from index 5 to 6, VOTER d9296c071f334a2c9078772294a56f25 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.626924 32333 tablet_service.cc:1514] Processing DeleteTablet for tablet 892ece785cc34f98a448bcac42c54788 with delete_type TABLET_DATA_TOMBSTONED (TS d9296c071f334a2c9078772294a56f25 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49098
I20240503 02:19:33.635406  1425 tablet_replica.cc:330] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25: stopping tablet replica
I20240503 02:19:33.635839  1425 raft_consensus.cc:2229] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:33.636256  1425 raft_consensus.cc:2258] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:33.637773  1425 ts_tablet_manager.cc:1875] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:33.642832  1425 ts_tablet_manager.cc:1888] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:19:33.642984  1425 log.cc:1196] T 892ece785cc34f98a448bcac42c54788 P d9296c071f334a2c9078772294a56f25: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/892ece785cc34f98a448bcac42c54788
I20240503 02:19:33.643961 32198 catalog_manager.cc:4830] TS d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943): tablet 892ece785cc34f98a448bcac42c54788 (table rebalance_test_table_2 [id=fe33a9ba0459484aabdfc05149cc3d53]) successfully deleted
I20240503 02:19:33.653888  1278 raft_consensus.cc:1050] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af: attempting to promote NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 to VOTER
I20240503 02:19:33.654592  1278 consensus_queue.cc:237] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } }
I20240503 02:19:33.657146   815 raft_consensus.cc:1261] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.657291 32355 raft_consensus.cc:1261] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.657783  1186 consensus_queue.cc:1035] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.658175  1392 consensus_queue.cc:1035] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.658463 32617 raft_consensus.cc:1261] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:33.658990  1186 consensus_queue.cc:1035] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:33.669493  1278 raft_consensus.cc:2932] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.670488   815 raft_consensus.cc:2932] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.672442 32355 raft_consensus.cc:2932] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.677109 32617 raft_consensus.cc:2932] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.677495 32211 catalog_manager.cc:5484] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9 reported cstate change: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.684041 32487 consensus_queue.cc:237] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } }
I20240503 02:19:33.686364 32355 raft_consensus.cc:1261] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:33.686957  1393 consensus_queue.cc:1035] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:33.687227   815 raft_consensus.cc:1261] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:33.687812  1393 consensus_queue.cc:1035] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:33.689620  1392 raft_consensus.cc:2932] T d90f1fe288904e879c20a7e9c9d7a140 P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.690532   815 raft_consensus.cc:2932] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.694567 32208 catalog_manager.cc:5484] T d90f1fe288904e879c20a7e9c9d7a140 P 8c3494a1fbad4d9aa0aa551169b78fa9 reported cstate change: config changed from index 5 to 6, VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.695201 32199 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d90f1fe288904e879c20a7e9c9d7a140 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:33.696187 32355 raft_consensus.cc:2932] T d90f1fe288904e879c20a7e9c9d7a140 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:33.699149 32598 tablet_service.cc:1514] Processing DeleteTablet for tablet d90f1fe288904e879c20a7e9c9d7a140 with delete_type TABLET_DATA_TOMBSTONED (TS 7f6faa7decfa4da2bceb8a4aff176d07 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39876
I20240503 02:19:33.699514  1412 tablet_replica.cc:330] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07: stopping tablet replica
I20240503 02:19:33.699714  1412 raft_consensus.cc:2229] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:33.699983  1412 raft_consensus.cc:2258] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:33.701884  1412 ts_tablet_manager.cc:1875] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:33.707787  1412 ts_tablet_manager.cc:1888] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:19:33.707901  1412 log.cc:1196] T d90f1fe288904e879c20a7e9c9d7a140 P 7f6faa7decfa4da2bceb8a4aff176d07: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/d90f1fe288904e879c20a7e9c9d7a140
I20240503 02:19:33.708676 32199 catalog_manager.cc:4830] TS 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283): tablet d90f1fe288904e879c20a7e9c9d7a140 (table rebalance_test_table_0 [id=3117cfc63bf2489389e6759839a19b7b]) successfully deleted
I20240503 02:19:34.011469  1460 ts_tablet_manager.cc:902] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9: Initiating tablet copy from peer 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993)
I20240503 02:19:34.012816  1460 tablet_copy_client.cc:308] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:41993
I20240503 02:19:34.013530 32507 tablet_copy_service.cc:140] P 37afeff9fa264bc69d4f70185f9665af: Received BeginTabletCopySession request for tablet 991ee5bce8494e908449e49097e14e45 from peer 8c3494a1fbad4d9aa0aa551169b78fa9 ({username='slave'} at 127.0.145.5:36803)
I20240503 02:19:34.013703 32507 tablet_copy_service.cc:161] P 37afeff9fa264bc69d4f70185f9665af: Beginning new tablet copy session on tablet 991ee5bce8494e908449e49097e14e45 from peer 8c3494a1fbad4d9aa0aa551169b78fa9 at {username='slave'} at 127.0.145.5:36803: session id = 8c3494a1fbad4d9aa0aa551169b78fa9-991ee5bce8494e908449e49097e14e45
I20240503 02:19:34.015715 32507 tablet_copy_source_session.cc:205] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:34.016917  1460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991ee5bce8494e908449e49097e14e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:34.021282  1460 tablet_copy_client.cc:783] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:34.021520  1460 tablet_copy_client.cc:647] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:34.023129  1460 tablet_copy_client.cc:523] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:34.025452  1460 tablet_bootstrap.cc:492] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap starting.
I20240503 02:19:34.059127  1460 tablet_bootstrap.cc:492] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:34.059619  1460 tablet_bootstrap.cc:492] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9: Bootstrap complete.
I20240503 02:19:34.059945  1460 ts_tablet_manager.cc:1367] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9: Time spent bootstrapping tablet: real 0.035s	user 0.035s	sys 0.000s
I20240503 02:19:34.060815  1460 raft_consensus.cc:348] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:34.061069  1460 raft_consensus.cc:729] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8c3494a1fbad4d9aa0aa551169b78fa9, State: Initialized, Role: LEARNER
I20240503 02:19:34.061234  1460 consensus_queue.cc:260] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: NON_VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: true } }
I20240503 02:19:34.061707  1460 ts_tablet_manager.cc:1398] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:19:34.062561 32507 tablet_copy_service.cc:342] P 37afeff9fa264bc69d4f70185f9665af: Request end of tablet copy session 8c3494a1fbad4d9aa0aa551169b78fa9-991ee5bce8494e908449e49097e14e45 received from {username='slave'} at 127.0.145.5:36803
I20240503 02:19:34.062736 32507 tablet_copy_service.cc:433] P 37afeff9fa264bc69d4f70185f9665af: ending tablet copy session 8c3494a1fbad4d9aa0aa551169b78fa9-991ee5bce8494e908449e49097e14e45 on tablet 991ee5bce8494e908449e49097e14e45 with peer 8c3494a1fbad4d9aa0aa551169b78fa9
I20240503 02:19:34.423564   815 raft_consensus.cc:1203] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:19:35.047278  1493 raft_consensus.cc:1050] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af: attempting to promote NON_VOTER 8c3494a1fbad4d9aa0aa551169b78fa9 to VOTER
I20240503 02:19:35.048122  1493 consensus_queue.cc:237] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } }
I20240503 02:19:35.050755 32617 raft_consensus.cc:1261] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:35.050750   815 raft_consensus.cc:1261] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 LEARNER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:35.051287 32355 raft_consensus.cc:1261] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:19:35.051347  1477 consensus_queue.cc:1035] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:35.051841  1495 consensus_queue.cc:1035] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:35.052179  1493 consensus_queue.cc:1035] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:19:35.055142  1495 raft_consensus.cc:2932] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:35.055738 32355 raft_consensus.cc:2932] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:35.056037   815 raft_consensus.cc:2932] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:35.060047 32208 catalog_manager.cc:5484] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25 reported cstate change: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:35.061439 32617 raft_consensus.cc:2932] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } attrs { replace: true } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:35.073503 32487 consensus_queue.cc:237] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } }
I20240503 02:19:35.075402   815 raft_consensus.cc:1261] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:35.075573 32617 raft_consensus.cc:1261] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:19:35.075999  1499 consensus_queue.cc:1035] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:35.076723  1499 consensus_queue.cc:1035] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:19:35.078352  1477 raft_consensus.cc:2932] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d9296c071f334a2c9078772294a56f25 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:35.078899   815 raft_consensus.cc:2932] T 991ee5bce8494e908449e49097e14e45 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d9296c071f334a2c9078772294a56f25 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:35.079546 32617 raft_consensus.cc:2932] T 991ee5bce8494e908449e49097e14e45 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d9296c071f334a2c9078772294a56f25 (127.0.145.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } }
I20240503 02:19:35.082007 32199 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 991ee5bce8494e908449e49097e14e45 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:35.084198 32211 catalog_manager.cc:5484] T 991ee5bce8494e908449e49097e14e45 P 37afeff9fa264bc69d4f70185f9665af reported cstate change: config changed from index 5 to 6, VOTER d9296c071f334a2c9078772294a56f25 (127.0.145.1) evicted. New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:35.087546 32333 tablet_service.cc:1514] Processing DeleteTablet for tablet 991ee5bce8494e908449e49097e14e45 with delete_type TABLET_DATA_TOMBSTONED (TS d9296c071f334a2c9078772294a56f25 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49098
I20240503 02:19:35.088426  1505 tablet_replica.cc:330] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25: stopping tablet replica
I20240503 02:19:35.088598  1505 raft_consensus.cc:2229] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:35.088827  1505 raft_consensus.cc:2258] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:35.089509  1505 ts_tablet_manager.cc:1875] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:35.093783  1505 ts_tablet_manager.cc:1888] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:19:35.093890  1505 log.cc:1196] T 991ee5bce8494e908449e49097e14e45 P d9296c071f334a2c9078772294a56f25: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-0/wal/wals/991ee5bce8494e908449e49097e14e45
I20240503 02:19:35.094659 32198 catalog_manager.cc:4830] TS d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943): tablet 991ee5bce8494e908449e49097e14e45 (table rebalance_test_table_1 [id=71ee030fc6ed4819be1bd2168142c3a4]) successfully deleted
I20240503 02:19:35.314338 32467 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:35.319582   796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:35.323374 32598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:35.339295   928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:35.352617 32333 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:35.506008 32467 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:35.515379 32598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:35.527328   928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:35.529027 32333 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:35.529659   796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:35.607606 32487 consensus_queue.cc:237] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: true } }
I20240503 02:19:35.609829   815 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:19:35.610462  1499 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:19:35.610896 32421 consensus_peers.cc:485] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af -> Peer 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283): Couldn't send request to peer 7f6faa7decfa4da2bceb8a4aff176d07. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:35.610905 32355 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:19:35.611510  1499 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:19:35.613741  1499 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: true } } }
I20240503 02:19:35.615864 32355 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: true } } }
I20240503 02:19:35.617990 32213 catalog_manager.cc:5484] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af reported cstate change: config changed from index 6 to 7, NON_VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) added. New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:35.619109   815 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: true } } }
I20240503 02:19:35.639005 32355 consensus_queue.cc:237] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: true } }
I20240503 02:19:35.641459   815 raft_consensus.cc:1261] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:19:35.641510 32617 raft_consensus.cc:1261] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:19:35.642062 32289 consensus_peers.cc:485] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 -> Peer 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993): Couldn't send request to peer 37afeff9fa264bc69d4f70185f9665af. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:19:35.642382   620 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:19:35.642702  1602 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:19:35.645303  1603 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: true } } }
I20240503 02:19:35.645897   815 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: true } } }
I20240503 02:19:35.647310 32617 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: true } } }
I20240503 02:19:35.650755 32212 catalog_manager.cc:5484] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 reported cstate change: config changed from index 6 to 7, NON_VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) added. New cstate: current_term: 1 leader_uuid: "d9296c071f334a2c9078772294a56f25" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: true } } }
I20240503 02:19:36.168453  1638 ts_tablet_manager.cc:902] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: Initiating tablet copy from peer 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993)
I20240503 02:19:36.169986  1638 tablet_copy_client.cc:308] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.2:41993
I20240503 02:19:36.170909 32507 tablet_copy_service.cc:140] P 37afeff9fa264bc69d4f70185f9665af: Received BeginTabletCopySession request for tablet 10541b8e86304eebbe61b7ebc176eb9d from peer 7f6faa7decfa4da2bceb8a4aff176d07 ({username='slave'} at 127.0.145.3:46149)
I20240503 02:19:36.171090 32507 tablet_copy_service.cc:161] P 37afeff9fa264bc69d4f70185f9665af: Beginning new tablet copy session on tablet 10541b8e86304eebbe61b7ebc176eb9d from peer 7f6faa7decfa4da2bceb8a4aff176d07 at {username='slave'} at 127.0.145.3:46149: session id = 7f6faa7decfa4da2bceb8a4aff176d07-10541b8e86304eebbe61b7ebc176eb9d
I20240503 02:19:36.173100 32507 tablet_copy_source_session.cc:205] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:36.174597  1638 ts_tablet_manager.cc:1875] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:19:36.179443  1638 ts_tablet_manager.cc:1888] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:19:36.179752  1638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10541b8e86304eebbe61b7ebc176eb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:36.182039  1638 tablet_copy_client.cc:783] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:36.182387  1638 tablet_copy_client.cc:647] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:36.184515  1638 tablet_copy_client.cc:523] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:36.187418  1638 tablet_bootstrap.cc:492] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: Bootstrap starting.
I20240503 02:19:36.219766  1638 tablet_bootstrap.cc:492] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:36.220278  1638 tablet_bootstrap.cc:492] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: Bootstrap complete.
I20240503 02:19:36.220556  1638 ts_tablet_manager.cc:1367] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent bootstrapping tablet: real 0.033s	user 0.032s	sys 0.000s
I20240503 02:19:36.221343  1638 raft_consensus.cc:348] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: true } }
I20240503 02:19:36.221597  1638 raft_consensus.cc:729] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7f6faa7decfa4da2bceb8a4aff176d07, State: Initialized, Role: LEARNER
I20240503 02:19:36.221804  1638 consensus_queue.cc:260] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: NON_VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: true } }
I20240503 02:19:36.222420  1638 ts_tablet_manager.cc:1398] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:19:36.223217 32507 tablet_copy_service.cc:342] P 37afeff9fa264bc69d4f70185f9665af: Request end of tablet copy session 7f6faa7decfa4da2bceb8a4aff176d07-10541b8e86304eebbe61b7ebc176eb9d received from {username='slave'} at 127.0.145.3:46149
I20240503 02:19:36.223356 32507 tablet_copy_service.cc:433] P 37afeff9fa264bc69d4f70185f9665af: ending tablet copy session 7f6faa7decfa4da2bceb8a4aff176d07-10541b8e86304eebbe61b7ebc176eb9d on tablet 10541b8e86304eebbe61b7ebc176eb9d with peer 7f6faa7decfa4da2bceb8a4aff176d07
I20240503 02:19:36.224092  1644 ts_tablet_manager.cc:902] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: Initiating tablet copy from peer d9296c071f334a2c9078772294a56f25 (127.0.145.1:45943)
I20240503 02:19:36.226410  1644 tablet_copy_client.cc:308] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.145.1:45943
I20240503 02:19:36.227253 32375 tablet_copy_service.cc:140] P d9296c071f334a2c9078772294a56f25: Received BeginTabletCopySession request for tablet 1e1f43f03ccf472c821943d848064d63 from peer 37afeff9fa264bc69d4f70185f9665af ({username='slave'} at 127.0.145.2:35853)
I20240503 02:19:36.227406 32375 tablet_copy_service.cc:161] P d9296c071f334a2c9078772294a56f25: Beginning new tablet copy session on tablet 1e1f43f03ccf472c821943d848064d63 from peer 37afeff9fa264bc69d4f70185f9665af at {username='slave'} at 127.0.145.2:35853: session id = 37afeff9fa264bc69d4f70185f9665af-1e1f43f03ccf472c821943d848064d63
I20240503 02:19:36.229194 32375 tablet_copy_source_session.cc:205] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:19:36.230440  1644 ts_tablet_manager.cc:1875] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:19:36.235098  1644 ts_tablet_manager.cc:1888] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:19:36.235415  1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1f43f03ccf472c821943d848064d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:19:36.237699  1644 tablet_copy_client.cc:783] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: tablet copy: Starting download of 0 data blocks...
I20240503 02:19:36.237998  1644 tablet_copy_client.cc:647] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: tablet copy: Starting download of 1 WAL segments...
I20240503 02:19:36.239885  1644 tablet_copy_client.cc:523] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:19:36.242566  1644 tablet_bootstrap.cc:492] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: Bootstrap starting.
I20240503 02:19:36.275295  1644 tablet_bootstrap.cc:492] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:19:36.275720  1644 tablet_bootstrap.cc:492] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: Bootstrap complete.
I20240503 02:19:36.275985  1644 ts_tablet_manager.cc:1367] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: Time spent bootstrapping tablet: real 0.034s	user 0.027s	sys 0.008s
I20240503 02:19:36.276659  1644 raft_consensus.cc:348] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: true } }
I20240503 02:19:36.276881  1644 raft_consensus.cc:729] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Initialized, Role: LEARNER
I20240503 02:19:36.277088  1644 consensus_queue.cc:260] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: NON_VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: true } }
I20240503 02:19:36.277669  1644 ts_tablet_manager.cc:1398] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:19:36.278702 32375 tablet_copy_service.cc:342] P d9296c071f334a2c9078772294a56f25: Request end of tablet copy session 37afeff9fa264bc69d4f70185f9665af-1e1f43f03ccf472c821943d848064d63 received from {username='slave'} at 127.0.145.2:35853
I20240503 02:19:36.278864 32375 tablet_copy_service.cc:433] P d9296c071f334a2c9078772294a56f25: ending tablet copy session 37afeff9fa264bc69d4f70185f9665af-1e1f43f03ccf472c821943d848064d63 on tablet 1e1f43f03ccf472c821943d848064d63 with peer 37afeff9fa264bc69d4f70185f9665af
I20240503 02:19:36.668694 32617 raft_consensus.cc:1203] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:19:36.752616 32487 raft_consensus.cc:1203] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:19:37.126076  1670 raft_consensus.cc:1050] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: attempting to promote NON_VOTER 7f6faa7decfa4da2bceb8a4aff176d07 to VOTER
I20240503 02:19:37.126920  1670 consensus_queue.cc:237] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } }
I20240503 02:19:37.129070 32617 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 LEARNER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:37.129071   815 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:37.129310 32355 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Refusing update from remote peer 37afeff9fa264bc69d4f70185f9665af: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:37.129848  1671 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:37.130206  1477 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:37.130740  1477 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:37.144886  1695 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } } }
I20240503 02:19:37.145774 32617 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } } }
I20240503 02:19:37.146073   815 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } } }
I20240503 02:19:37.148746 32355 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } } }
I20240503 02:19:37.149873 32211 catalog_manager.cc:5484] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af reported cstate change: config changed from index 7 to 8, 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "37afeff9fa264bc69d4f70185f9665af" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:37.155190  1672 raft_consensus.cc:1050] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25: attempting to promote NON_VOTER 37afeff9fa264bc69d4f70185f9665af to VOTER
I20240503 02:19:37.155980  1672 consensus_queue.cc:237] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: false } }
I20240503 02:19:37.158828   815 raft_consensus.cc:1261] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:37.158849 32487 raft_consensus.cc:1261] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 1 LEARNER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:37.158962 32617 raft_consensus.cc:1261] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:19:37.159432   620 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:37.159761  1673 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:37.160215   620 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:19:37.170310  1704 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: false } } }
I20240503 02:19:37.171049 32617 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: false } } }
I20240503 02:19:37.176221 32487 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: false } } }
I20240503 02:19:37.180696   815 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: false } } }
I20240503 02:19:37.181264 32213 catalog_manager.cc:5484] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 reported cstate change: config changed from index 7 to 8, 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d9296c071f334a2c9078772294a56f25" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { replace: true } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: false } } }
I20240503 02:19:37.190515 32355 consensus_queue.cc:237] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: false } }
I20240503 02:19:37.192348 32487 raft_consensus.cc:1261] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:19:37.192889  1704 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:19:37.193781   815 raft_consensus.cc:1261] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:19:37.194298  1704 consensus_queue.cc:1035] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:19:37.195900   620 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: false } } }
I20240503 02:19:37.196972 32487 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: false } } }
I20240503 02:19:37.199244 32198 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1e1f43f03ccf472c821943d848064d63 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:37.199702   815 raft_consensus.cc:2932] T 1e1f43f03ccf472c821943d848064d63 P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: false } } }
I20240503 02:19:37.201862 32208 catalog_manager.cc:5484] T 1e1f43f03ccf472c821943d848064d63 P d9296c071f334a2c9078772294a56f25 reported cstate change: config changed from index 8 to 9, VOTER 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3) evicted. New cstate: current_term: 1 leader_uuid: "d9296c071f334a2c9078772294a56f25" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:37.207592 32598 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e1f43f03ccf472c821943d848064d63 with delete_type TABLET_DATA_TOMBSTONED (TS 7f6faa7decfa4da2bceb8a4aff176d07 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39876
I20240503 02:19:37.208361  1711 tablet_replica.cc:330] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07: stopping tablet replica
I20240503 02:19:37.208644  1711 raft_consensus.cc:2229] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:37.209754  1711 raft_consensus.cc:2258] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:37.210659  1711 ts_tablet_manager.cc:1875] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:37.216898  1711 ts_tablet_manager.cc:1888] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:19:37.217013  1711 log.cc:1196] T 1e1f43f03ccf472c821943d848064d63 P 7f6faa7decfa4da2bceb8a4aff176d07: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-2/wal/wals/1e1f43f03ccf472c821943d848064d63
I20240503 02:19:37.217862 32199 catalog_manager.cc:4830] TS 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283): tablet 1e1f43f03ccf472c821943d848064d63 (table rebalance_test_table_1 [id=71ee030fc6ed4819be1bd2168142c3a4]) successfully deleted
I20240503 02:19:37.232952 32487 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10541b8e86304eebbe61b7ebc176eb9d"
dest_uuid: "37afeff9fa264bc69d4f70185f9665af"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53322
I20240503 02:19:37.233196 32487 raft_consensus.cc:595] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Received request to transfer leadership
I20240503 02:19:37.250370 32487 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10541b8e86304eebbe61b7ebc176eb9d"
dest_uuid: "37afeff9fa264bc69d4f70185f9665af"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53348
I20240503 02:19:37.250552 32487 raft_consensus.cc:595] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Received request to transfer leadership
I20240503 02:19:37.468255 32487 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10541b8e86304eebbe61b7ebc176eb9d"
dest_uuid: "37afeff9fa264bc69d4f70185f9665af"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53358
I20240503 02:19:37.468442 32487 raft_consensus.cc:595] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Received request to transfer leadership
I20240503 02:19:37.542778  1695 raft_consensus.cc:979] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: : Instructing follower d9296c071f334a2c9078772294a56f25 to start an election
I20240503 02:19:37.542977  1695 raft_consensus.cc:1067] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Signalling peer d9296c071f334a2c9078772294a56f25 to start an election
I20240503 02:19:37.543938 32355 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10541b8e86304eebbe61b7ebc176eb9d"
dest_uuid: "d9296c071f334a2c9078772294a56f25"
 from {username='slave'} at 127.0.145.2:35853
I20240503 02:19:37.544175 32355 raft_consensus.cc:482] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:19:37.544307 32355 raft_consensus.cc:3037] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:19:37.546175 32355 raft_consensus.cc:504] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } }
I20240503 02:19:37.547037 32355 leader_election.cc:288] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [CANDIDATE]: Term 2 election: Requested vote from peers 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993), 8c3494a1fbad4d9aa0aa551169b78fa9 (127.0.145.5:41015), 7f6faa7decfa4da2bceb8a4aff176d07 (127.0.145.3:38283)
I20240503 02:19:37.547329 32487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10541b8e86304eebbe61b7ebc176eb9d" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "37afeff9fa264bc69d4f70185f9665af"
I20240503 02:19:37.547550 32487 raft_consensus.cc:3032] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:19:37.547477   815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10541b8e86304eebbe61b7ebc176eb9d" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9"
I20240503 02:19:37.547645 32487 raft_consensus.cc:729] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 37afeff9fa264bc69d4f70185f9665af, State: Running, Role: LEADER
I20240503 02:19:37.547792 32617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10541b8e86304eebbe61b7ebc176eb9d" candidate_uuid: "d9296c071f334a2c9078772294a56f25" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7f6faa7decfa4da2bceb8a4aff176d07"
I20240503 02:19:37.547824   815 raft_consensus.cc:3037] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:19:37.547904 32487 consensus_queue.cc:260] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } }
I20240503 02:19:37.548007 32617 raft_consensus.cc:3037] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:19:37.548228 32487 raft_consensus.cc:3037] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:19:37.549957 32487 raft_consensus.cc:2454] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 2.
I20240503 02:19:37.550791 32617 raft_consensus.cc:2454] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 2.
I20240503 02:19:37.550798   815 raft_consensus.cc:2454] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9296c071f334a2c9078772294a56f25 in term 2.
I20240503 02:19:37.551311 32289 leader_election.cc:302] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 4 yes votes; 0 no votes. yes voters: 37afeff9fa264bc69d4f70185f9665af, 7f6faa7decfa4da2bceb8a4aff176d07, 8c3494a1fbad4d9aa0aa551169b78fa9, d9296c071f334a2c9078772294a56f25; no voters: 
I20240503 02:19:37.551669  1704 raft_consensus.cc:2781] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:19:37.551842  1704 raft_consensus.cc:686] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 2 LEADER]: Becoming Leader. State: Replica: d9296c071f334a2c9078772294a56f25, State: Running, Role: LEADER
I20240503 02:19:37.552125  1704 consensus_queue.cc:237] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } }
I20240503 02:19:37.554767 32208 catalog_manager.cc:5484] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 reported cstate change: term changed from 1 to 2, leader changed from 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) to d9296c071f334a2c9078772294a56f25 (127.0.145.1). New cstate: current_term: 2 leader_uuid: "d9296c071f334a2c9078772294a56f25" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:19:37.996754 32487 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 2 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:19:37.997434  1704 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37afeff9fa264bc69d4f70185f9665af" member_type: VOTER last_known_addr { host: "127.0.145.2" port: 41993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:19:38.035804 32617 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 2 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:19:38.036353  1704 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:19:38.040838   815 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 2 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:19:38.041460  1704 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:19:38.503289 32355 consensus_queue.cc:237] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } }
I20240503 02:19:38.505013   815 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 2 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:19:38.505012 32617 raft_consensus.cc:1261] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 2 FOLLOWER]: Refusing update from remote peer d9296c071f334a2c9078772294a56f25: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:19:38.505637  1756 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:19:38.505967  1704 consensus_queue.cc:1035] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:19:38.508234  1756 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } } }
I20240503 02:19:38.508952 32617 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P 7f6faa7decfa4da2bceb8a4aff176d07 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } } }
I20240503 02:19:38.509152   815 raft_consensus.cc:2932] T 10541b8e86304eebbe61b7ebc176eb9d P 8c3494a1fbad4d9aa0aa551169b78fa9 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } } }
I20240503 02:19:38.512264 32198 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 10541b8e86304eebbe61b7ebc176eb9d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:19:38.513927 32208 catalog_manager.cc:5484] T 10541b8e86304eebbe61b7ebc176eb9d P d9296c071f334a2c9078772294a56f25 reported cstate change: config changed from index 8 to 10, VOTER 37afeff9fa264bc69d4f70185f9665af (127.0.145.2) evicted. New cstate: current_term: 2 leader_uuid: "d9296c071f334a2c9078772294a56f25" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d9296c071f334a2c9078772294a56f25" member_type: VOTER last_known_addr { host: "127.0.145.1" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3494a1fbad4d9aa0aa551169b78fa9" member_type: VOTER last_known_addr { host: "127.0.145.5" port: 41015 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f6faa7decfa4da2bceb8a4aff176d07" member_type: VOTER last_known_addr { host: "127.0.145.3" port: 38283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:19:38.522070 32467 tablet_service.cc:1514] Processing DeleteTablet for tablet 10541b8e86304eebbe61b7ebc176eb9d with delete_type TABLET_DATA_TOMBSTONED (TS 37afeff9fa264bc69d4f70185f9665af not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:58692
I20240503 02:19:38.525074  1781 tablet_replica.cc:330] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: stopping tablet replica
I20240503 02:19:38.525276  1781 raft_consensus.cc:2229] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:19:38.525472  1781 raft_consensus.cc:2258] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:19:38.526757  1781 ts_tablet_manager.cc:1875] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:19:38.532889  1781 ts_tablet_manager.cc:1888] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:19:38.533119  1781 log.cc:1196] T 10541b8e86304eebbe61b7ebc176eb9d P 37afeff9fa264bc69d4f70185f9665af: Deleting WAL directory at /tmp/dist-test-taskfpuZdt/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714702277455189-580-0/raft_consensus-itest-cluster/ts-1/wal/wals/10541b8e86304eebbe61b7ebc176eb9d
I20240503 02:19:38.538996 32199 catalog_manager.cc:4830] TS 37afeff9fa264bc69d4f70185f9665af (127.0.145.2:41993): tablet 10541b8e86304eebbe61b7ebc176eb9d (table rebalance_test_table_0 [id=3117cfc63bf2489389e6759839a19b7b]) successfully deleted
I20240503 02:19:38.644357   796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:38.649649 32467 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:38.663853 32598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:38.667117 32333 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:38.668447   928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:38.817481   796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:38.821574 32598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:38.828091 32467 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:38.830677 32333 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:38.832819   928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:39.446301 32208 ts_manager.cc:292] Set tserver state for b5704d091d784c849a96fe720e53cc45 to MAINTENANCE_MODE
I20240503 02:19:40.947940 32598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:40.948343 32467 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:40.957221   796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:40.989229 32333 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:40.994828   928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:41.165191   796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:41.165670 32598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:41.169451 32467 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:41.178452   928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:41.192411 32333 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:41.369277 32467 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:41.373555   796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:41.374572 32598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:41.377171   928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:41.380025 32333 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:41.567005   796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:41.571514 32598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:41.572957   928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:41.573812 32467 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:41.601945 32333 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:41.748587   796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:41.750133 32467 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:41.756724   928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:41.758966 32333 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:41.766758 32598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:41.945986 32467 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:41.948892   796 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:41.949991 32598 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:41.961112 32333 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:19:41.962226   928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:42.149135   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 32273
I20240503 02:19:42.161448 32395 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:42.304926   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 32273
W20240503 02:19:42.331400   752 connection.cc:535] server connection from 127.0.145.1:52025 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:19:42.332345   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 32404
I20240503 02:19:42.337795 32526 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:42.483299   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 32404
I20240503 02:19:42.510021   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 32536
I20240503 02:19:42.516256 32657 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:42.654109   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 32536
W20240503 02:19:42.679250 32201 connection.cc:535] server connection from 127.0.145.3:54165 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:19:42.679566   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 865
I20240503 02:19:42.685748   988 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:42.792471   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 865
W20240503 02:19:42.813009 32198 connection.cc:535] server connection from 127.0.145.4:49535 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:19:42.813215   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 734
I20240503 02:19:42.819295   855 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:42.947207   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 734
I20240503 02:19:42.972518   580 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu pid 32183
I20240503 02:19:42.973613 32244 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:19:43.122959   580 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfpuZdt/build/asan/bin/kudu with pid 32183
2024-05-03T02:19:43Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2 (23629 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (46997 ms total)

[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (505704 ms total)
[  PASSED  ] 6 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00>

 1 FAILED TEST